Scene motion correction in fused image systems

Patent metadata

Field	Value
Publication number	US-9843730-B2
Application number	US-201715587617-A
Country	US
Kind code	B2
Filing date	May 5, 2017
Priority date	May 30, 2014
Publication date	Dec 12, 2017
Grant date	Dec 12, 2017

Abstract

Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

First claim

The invention claimed is:

1. A method to fuse images based on scene motion, comprising:
obtaining a plurality of images including at least one short-exposure image of a scene and at least one long-exposure image of the scene;
determining a motion value indicative of an amount of motion in the scene;
designating one of the at least one long-exposure image as an output image when the motion value is less than a first threshold; and
designating a combined image as the output image when the motion value is greater than or equal to the first threshold,
wherein the combined image is formed from a combination of a plurality of the at least one short-exposure image and the at least one long-exposure image.

2. The method of claim 1 , wherein determining a motion value indicative of an amount of motion in the scene is based on a first spatial difference map.

3. The method of claim 2 , wherein the first spatial difference map is determined based on comparing a first short-exposure image with a second short-exposure image.

4. The method of claim 3 , further comprising:
determining each pixel of the first spatial difference map based on one or more pixels of the first short-exposure image and a corresponding one or more pixels of the second short-exposure image.

5. The method of claim 4 , wherein designating a combined image as an output image further comprises:
determining a reduced-noise short-exposure image based on the first and the second short-exposure images and the first spatial difference map, and
determining the combined image based on a combination of the reduced-noise short-exposure image and one of the at least one long-exposure image.

6. The method of claim 5 , wherein determining the combined image further comprises:
combining the reduced-noise short-exposure image and one of the at least one long-exposure image to generate an intermediate output image; and
filtering the intermediate output image to generate the combined image.

7. The method of claim 5 , wherein determining the combined image further comprises:
determining a second spatial difference map based on the reduced-noise short-exposure image and one of the at least one long-exposure image; and
determining the combined image based on a weighted combination of the reduced-noise short-exposure image and the one of the at least one long-exposure image, wherein the weighted combination is based on the second spatial difference map.

8. An electronic device, comprising:
an image capture system;
a display unit;
a memory coupled to the image capture system and the display unit;
one or more processors coupled to the image capture system, the display unit and the memory, the one or more processors configured to execute instructions stored in the memory to
obtain a plurality of images including at least one short-exposure image of a scene and at least one long-exposure image of the scene;
determine a motion value indicative of an amount of motion in the scene;
designate one of the at least one long-exposure image as an output image when the motion value is less than a first threshold; and
designate a combined image as the output image when the motion value is greater than or equal to the first threshold,
wherein the combined image is formed from a combination of a plurality of the at least one short-exposure image and the at least one long-exposure image.

9. The electronic device of claim 8 , wherein the instructions to cause the one or more processors to determine a motion value indicative of an amount of motion in the scene comprise instructions to cause the one or more processors to determine a motion value indicative of an amount of motion in the scene based on a first spatial difference map.

10. The electronic device of claim 9 , wherein the first spatial difference map is determined based on comparing a first short-exposure image with a second short-exposure image.

11. The electronic device of claim 10 , wherein the one or more processors are further configured to execute instructions stored in the memory to:
determine each pixel of the first spatial difference map based on one or more pixels of the first short-exposure image and a corresponding one or more pixels of the second short-exposure image.

12. The electronic device of claim 11 , wherein the instructions to cause one or more processors to designate a combined image as an output image further comprise instructions to cause the one or more processors to:
determine a reduced-noise short-exposure image based on the first and the second short-exposure images and the first spatial difference map, and
determine the combined image based on a combination of the reduced-noise short-exposure image and one of the at least one long-exposure image.

13. The electronic device of claim 12 , wherein the instructions to cause one or more processors to determine the combined image further comprise instructions to cause the one or more processors to:
combine the reduced-noise short-exposure image and one of the at least one long-exposure image to generate an intermediate output image; and
filter the intermediate output image to generate the combined image.

14. A non-transitory program storage device comprising instructions stored thereon, the instructions readable by one or more processors and configured to cause one or more processors to:
obtain a plurality of images including at least one short-exposure image of a scene and at least one long-exposure image of the scene;
determine a motion value indicative of an amount of motion in the scene;
designate one of the at least one long-exposure image as an output image when the motion value is less than a first threshold; and
designate a combined image as the output image when the motion value is greater than or equal to the first threshold,
wherein the combined image is formed from a combination of a plurality of the at least one short-exposure image and the at least one long-exposure image.

15. The non-transitory program storage device of claim 14 , wherein the instructions to cause the one or more processors to determine a motion value indicative of an amount of motion in the scene comprise instructions to cause the one or more processors to determine a motion value indicative of an amount of motion in the scene based on a first spatial difference map.

16. The non-transitory program storage device of claim 15 , wherein the first spatial difference map is determined based on comparing a first short-exposure image with a second short-exposure image.

17. The non-transitory program storage device of claim 16 , wherein the instructions readable by one or more processors are configured to further cause one or more processors to:
determine each pixel of the first spatial difference map based on one or more pixels of the first short-exposure image and a corresponding one or more pixels of the second short-exposure image.

18. The non-transitory program storage device of claim 17 , wherein the instructions to cause one or more processors to designate a combined image as an output image further comprise instructions to cause the one or more processors to:
determine a reduced-noise short-exposure image based on the first and the second short-exposure images and the first spatial difference map, and
determine the combined image based on a combination of the reduced-noise short-exposure image and one of the at least one long-exposure image.

19. The non-transitory program storage device of claim 18 , wherein the instructions to cause one or more processors

Assignees

• Apple Inc

Inventors

• Molgaard Claus
• Tico Marius
• Toft Rolf
• Hubel Paul M

Classifications

• H04N23/6811

  based on the image signal · CPC title

• H04N23/6845 Primary

  by combination of a plurality of images sequentially taken · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N25/589

  with different integration times, e.g. short and long exposures · CPC title