Scene motion correction in fused image systems

Patent metadata

Field	Value
Publication number	US-9342871-B2
Application number	US-201414502887-A
Country	US
Kind code	B2
Filing date	Sep 30, 2014
Priority date	May 30, 2014
Publication date	May 17, 2016
Grant date	May 17, 2016

Abstract

Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

First claim

The invention claimed is:

1. A method to fuse images based on scene motion, comprising:
obtaining a first short-exposure image of a scene followed immediately by a long-exposure image of the scene followed immediately by a second short-exposure image of the scene;
determining a first spatial difference map based on the first and second short-exposure images;
determining a motion value indicative of an amount of motion of an object in the scene based on the first spatial difference map;
designating the long-exposure image as an output image when the motion value is less than a first threshold or greater than a second threshold;
determining the output image in accordance with the following when the motion value is greater than or equal to the first threshold and less than or equal to the second threshold—
determining a reduced-noise short-exposure image based on the first and second short-exposure images and the first spatial difference map, and
determining the output image based on a combination of the reduced-noise short-exposure image and the long-exposure image; and

storing the output image in a memory.

2. The method of claim 1 , wherein determining a motion value indicative of an amount of motion of an object in the scene based on the first spatial difference map comprises determining a value of one or more elements of the first spatial difference map is greater than a specified threshold.

3. The method of claim 1 , wherein determining a first spatial difference map comprises determining each pixel of the first spatial difference map based on one or more pixels of the first short-exposure image and a corresponding one or more pixels of the second short-exposure image.

4. The method of claim 3 , wherein determining the output image based on a combination of the reduced-noise short-exposure image and the long-exposure image comprises:
combining the reduced-noise short-exposure image and the long-exposure image to generate an intermediate output image; and
filtering the intermediate output image to generate the output image.

5. The method of claim 1 , wherein determining the output image based on a combination of the reduced-noise short-exposure image and the long-exposure image comprises:
determining a second spatial difference map based on the reduced-noise short-exposure image and the long-exposure image; and
determining the output image based on a weighted combination of the reduced-noise short-exposure image and the long-exposure image, wherein the weighted combination is based on the second spatial difference map.

6. An electronic device, comprising:
an image capture system;
a display unit;
a memory coupled to the image capture system and the display unit;
one or more processors coupled to the image capture system, the display unit and the memory, the one or more processors configured to execute instructions stored in the memory to—
obtain a first short-exposure image of a scene followed immediately by a long-exposure image of the scene followed immediately by a second short-exposure image of the scene,
determine a first spatial difference map based on the first and second short-exposure images,
determine a motion value indicative of an amount of motion of an object in the scene based on the first spatial difference map,
designate the long-exposure image as an output image when the motion value is less than a first threshold or greater than a second threshold,
determine the output image in accordance with the following when the motion value is greater than or equal to the first threshold and less than or equal to the second threshold—
determine a reduced-noise short-exposure image based on the first and second short-exposure images and the first spatial difference map, and
determine the output image based on a combination of the reduced-noise short-exposure image and the long-exposure image; and

store the output image in the memory coupled to the display unit.

7. The electronic device of claim 6 , wherein the instructions to cause the one or more processors to determine a motion value indicative of an amount of motion of an object in the scene based on the first spatial difference map comprise instructions to cause the one or more processors to determine a value of one or more elements of the first spatial difference map is greater than a specified threshold.

8. The electronic device of claim 6 , wherein the instructions to cause the one or more processors to determine a first spatial difference map comprise instructions to cause the one or more processors to determine each pixel of the first spatial difference map based on one or more pixels of the first short-exposure image and a corresponding one or more pixels of the second short-exposure image.

9. The electronic device of claim 8 , wherein the instructions to cause the one or more processors to determine the output image comprise instructions to cause the one or more processors to:
combine the reduced-noise short-exposure image and the long-exposure image to generate an intermediate output image; and
filter the intermediate output image to generate the output image.

10. The electronic device of claim 6 , wherein the instructions to cause the one or more processors to determine the output image based on a combination of the reduced-noise short-exposure image and the long-exposure image comprise instructions to cause the one or more processors to:
determine a second spatial difference map based on the reduced-noise short-exposure image and the long-exposure image; and
determine the output image based on a weighted combination of the reduced-noise short-exposure image and the long-exposure image, wherein the weighted combination is based on the second spatial difference map.

11. A non-transitory program storage device comprising instructions stored thereon, the instructions readable by one or more processors and configured to cause one or more processors to:
obtain a first long-exposure image of a scene followed immediately by a short-exposure image of the scene followed immediately by a second long-exposure image of the scene, each image having pixels wherein each pixel in the short-exposure image has a corresponding pixel in the first and second long-exposure images;
determine a first spatial difference map based on the first long-exposure image and the short-exposure image;
combine the first long-exposure image and the short-exposure image based on the first spatial difference map to generate a first intermediate image;
determine a second spatial difference map based on the second long-exposure image and the short-exposure image;
combine the second long-exposure image and the short-exposure image based on the second spatial difference map to generate a second intermediate image;
combine the first intermediate image and the second intermediate image to generate an output image; and
store the output image in a memory.

12. The non-transitory program storage device of claim 11 , wherein the instructions to generate an output image comprise instructions to cause the one or more processors to:
generate a third spatial difference map based on the first and second intermediate images;
combine the first and second intermediate images based on the third spatial difference map to generate the output image.

13. The non-transitory program storage device of claim 12 , wherein:
the instructions to cause the one or more processors to determine a first spatial difference map comprise instructions to cause the one or more processors to determine each pixel of the first spatial difference map based on one or more pixels of the first long-exposure image and a corr

Assignees

• Apple Inc

Inventors

• Molgaard Claus
• Tico Marius
• Toft Rolf
• Hubel Paul M

Classifications

• H04N25/589

  with different integration times, e.g. short and long exposures · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/6845 Primary

  by combination of a plurality of images sequentially taken · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/6811

  based on the image signal · CPC title