Scene motion correction in fused image systems

Patent metadata

Field	Value
Publication number	US-10277820-B2
Application number	US-201816030632-A
Country	US
Kind code	B2
Filing date	Jul 9, 2018
Priority date	May 30, 2014
Publication date	Apr 30, 2019
Grant date	Apr 30, 2019

Abstract

Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short-and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

First claim

The invention claimed is:

1. An electronic device, comprising:
an image capture device;
a memory operatively coupled to the image capture device; and
one or more processors operatively coupled to the memory and configured to execute instructions stored in the memory, the instructions configured to cause the one or more processors to:
capture, by the image capture device, one or more first images of a scene, wherein each first image has a first exposure time and comprises pixels;
store the one or more first images in the memory;
capture, by the image capture device, one or more second images of the scene, wherein each second image has a second exposure time shorter than the first exposure time, and comprises pixels;
store the one or more second images in the memory;
generate a spatial difference map, based, at least in part, on the first and second images, wherein the spatial difference map has values, and wherein each value in the spatial difference map has a corresponding pixel in each of the first and second images and is representative of an amount of motion in the scene at a location of the respective corresponding pixels;
determine a weight mask based, at least in part, on the one or more first images, the one or more second images, and the spatial difference map;
fuse the first and second images based, at least in part, on the spatial difference map and the weight mask; and
generate an output image based, at least in part, on the fused first and second images.

2. The electronic device of claim 1 , wherein the instructions further comprise instructions to cause the one or more processors to:
store the output image in the memory.

3. The electronic device of claim 1 , wherein the one or more first images comprise long-exposure images.

4. The electronic device of claim 1 , wherein the one or more second images comprise short-exposure images.

5. The electronic device of claim 1 , wherein the instructions to cause the one or more processors to generate a spatial difference map further comprise instructions to apply a first threshold to each value in the spatial difference map to convert each value in the spatial difference map to a binary value.

6. The electronic device of claim 1 , wherein the instructions to cause the one or more processors to generate a spatial difference map comprise instructions to cause the one or more processors to generate a spatial difference map in accordance with an optical flow analysis.

7. The electronic device of claim 1 , wherein the instructions to cause the one or more processors to generate the output image further comprise instructions to cause the one or more processors to:
generate an intermediate output image based on the fused first and second images and the weight mask.

8. The electronic device of claim 7 , further comprising instructions to cause the one or more processors to filter the intermediate output image.

9. The electronic device of claim 8 , wherein the instructions to cause the one or more processors to filter the intermediate output image comprise instructions to cause the one or more processors to filter each pixel in the intermediate output image in a manner that is inversely proportional to an amount of noise in the respective pixel.

10. A method, comprising:
obtaining one or more first images of a scene, wherein each first image has a first exposure time and comprises pixels;
storing the one or more first images in a memory;
obtaining one or more second images of the scene, wherein each second image has a second exposure time shorter than the first exposure time, and comprises pixels;
storing the one or more second images in the memory;
generating a spatial difference map, based, at least in part, on the first and second images, wherein the spatial difference map has values, and wherein each value in the spatial difference map has a corresponding pixel in each of the first and second images and is representative of an amount of motion in the scene at a location of the respective corresponding pixels;
determining a weight mask based, at least in part, on the one or more first images, the one or more second images, and the spatial difference map;
fusing the first and second images based, at least in part, on the spatial difference map and the weight mask; and
generating an output image based, at least in part, on the fused first and second images.

11. The method of claim 10 , wherein the one or more first images comprise long-exposure images, and wherein the one or more second images comprise short-exposure images.

12. The method of claim 10 , wherein generating a spatial difference map further comprises applying a first threshold to each value in the spatial difference map to convert each value in the spatial difference map to a binary value.

13. The method of claim 10 , wherein generating a spatial difference map further comprises generating a spatial difference map in accordance with an optical flow analysis.

14. The method of claim 10 , wherein generating the output image further comprises:
generating an intermediate output image based on the fused first and second images and the weight mask; and
filtering each pixel in the intermediate output image in a manner that is inversely proportional to an amount of noise in the respective pixel.

15. A non-transitory program storage device comprising instructions stored thereon, the instructions readable by one or more processors and configured to cause one or more processors to:
obtain one or more first images of a scene, wherein each first image has a first exposure time and comprises pixels;
store the one or more first images in a memory;
obtain one or more second images of the scene, wherein each second image has a second exposure time shorter than the first exposure time, and comprises pixels;
store the one or more second images in the memory;
generate a spatial difference map, based, at least in part, on the first and second images, wherein the spatial difference map has values, and wherein each value in the spatial difference map has a corresponding pixel in each of the first and second images and is representative of an amount of motion in the scene at a location of the respective corresponding pixels;
determine a weight mask based, at least in part, on the one or more first images, the one or more second images, and the spatial difference map;
fuse the first and second images based, at least in part, on the spatial difference map and the weight mask; and
generate an output image based, at least in part, on the fused first and second images.

16. The non-transitory program storage device of claim 15 , wherein the one or more first images comprise long-exposure images, and wherein the one or more second images comprise short-exposure images.

17. The non-transitory program storage device of claim 15 , wherein the instructions to cause the one or more processors to generate a spatial difference map further comprise instructions to apply a first threshold to each value in the spatial difference map to convert each value in the spatial difference map to a binary value.

18. The non-transitory program storage device of claim 15 , wherein the instructions to cause the one or more processors to generate a spatial difference map comprise instructions to cause the one or more processors to generate a spatial difference map in accordance with an optical flow analysis.

19. The non-transitory program storage device of claim 15 , wherein the instructions to cause the one or more processors to generate the o

Assignees

• Apple Inc

Inventors

• Molgaard Claus
• Tico Marius
• Toft Rolf
• Hubel Paul M

Classifications

• H04N23/6811

  based on the image signal · CPC title

• H04N25/626

  Reduction of noise due to residual charges remaining after image readout, e.g. to remove ghost images or afterimages · CPC title

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N25/589

  with different integration times, e.g. short and long exposures · CPC title