Reference frame selection for still image stabilization

Patent metadata

Field	Value
Publication number	US-9491360-B2
Application number	US-201313911873-A
Country	US
Kind code	B2
Filing date	Jun 6, 2013
Priority date	Jun 6, 2013
Publication date	Nov 8, 2016
Grant date	Nov 8, 2016

Abstract

Systems, methods, and computer readable media to improve image stabilization operations are described. A novel combination of image quality and commonality metrics are used to identify a reference frame from a set of commonly captured images which, when the set's other images are combined with it, results in a quality stabilized image. The disclosed image quality and commonality metrics may also be used to optimize the use of a limited amount of image buffer memory during image capture sequences that return more images that the memory may accommodate at one time. Image quality and commonality metrics may also be used to effect the combination of multiple relatively long-exposure images which, when combined with a one or more final (relatively) short-exposure images, yields images exhibiting motion-induced blurring in interesting and visually pleasing ways.

First claim

The invention claimed is:

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:
obtain a temporal sequence of images of a scene;
determine, for each of the obtained images, a quality metric value;
identify, from the obtained images, a set of relevant images based at least in part on the obtained images' quality metric values;
generate, from the relevant set of images, a temporary image;
determine, for each image in the set of relevant images, a commonality metric value between the each image and the temporary image;
identify, from the set of relevant images, a reference image based at least in part on the commonality metric values; and
combine one or more of the set of relevant images with the reference image to generate an output image of the scene.

2. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a quality metric value comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image's sharpness.

3. The non-transitory program storage device of claim 2 , wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image's sharpness comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on output from at least one of an auto-focus system and an auto-exposure system.

4. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a quality metric value comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a blur value based at least in part on output from one or more motion sensors.

5. The non-transitory program storage device of claim 2 , wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image's sharpness further comprise instructions to incorporate a secondary consideration.

6. The non-transitory program storage device of claim 5 , wherein the secondary consideration comprises output from an object detection system.

7. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to identify, from the obtained images, a set of relevant images comprise instructions to cause the one or more processors to:
identify a highest quality metric value from the obtained images' quality metric values;
determine, for each of the obtained images, a relative quality metric value based on a ratio of the image's quality metric value and the highest quality metric value; and
identify, as a relevant image, all those images from the obtained images having a relative quality metric value greater than a threshold value.

8. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to generate a temporary image comprise instructions to cause the one or more processors to determine an average image based, at least in part, on each of the relevant images.

9. The non-transitory program storage device of claim 8 , wherein the average image comprises a pixel-by-pixel average of each of the relevant images.

10. The non-transitory program storage device of claim 8 , wherein the instructions to cause the one or more processors to determine, for each of the relevant images, a commonality metric value comprise instructions to cause the one or more processors to determine a difference value between each of the relevant images and the average image, wherein the difference value represents a commonality metric value.

11. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to obtain a temporal sequence of images of a scene comprise instructions to cause the one or more processors to:
obtain an initial temporal sequence of images that includes N images;
determine, for each of the N images, a quality metric value; and
retain, as a temporal sequence of images, M of the N images based on the determined quality metric value for each of the N images, where M is less than N.

12. The non-transitory program storage device of claim 11 , wherein the instructions to cause the one or more processors to determine, for each of the N images, a quality metric value further comprise instructions to cause the one or more processors to determine, for each of the N images, a similarity metric value.

13. The non-transitory program storage device of claim 12 , wherein the instructions to cause the one or more processors to retain, as a temporal sequence of images, M of the N images based on the determined quality metric value for each of the N images comprise instructions to cause the one or more processors to retain, as a temporal sequence of images, M of the N images based on the determined quality metric value and the determined similarity metric value for each of the N images.

14. The non-transitory program storage device of claim 13 , wherein the similarity metric value between two images is based, at least in part, on a time between when the two images were obtained.

15. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to obtain a temporal sequence of images of a scene comprise instructions to cause the one or more programmable control devices to:
obtain an initial sequence of images, each image having an exposure time; and
obtain one or more final images after the initial sequence of images is obtained, wherein each of the one or more final images has an exposure time that is shorter than any exposure time of any image in the initial sequence of images.

16. The non-transitory program storage device of claim 15 , wherein the instructions to cause the one or more programmable control devices to identify, from the obtained images, a set of relevant images comprise instructions to cause the one or more programmable control devices to:
determine a first set of relevant images from the initial sequence of images based, at least in part, on the initial sequence of images' quality metric values; and
determine a second set of relevant images from the one or more final images based, at least in part, on the one or more final images' quality metric values.

17. The non-transitory program storage device of claim 16 , wherein the instructions to cause the one or more programmable control devices to determine, for each image in the set of relevant images, a commonality metric value comprise instructions to cause the one or more programmable control devices to:
determine, for each image in the first set of relevant images, a commonality metric value exclusive of the images in the second set of relevant images; and
determine, for each image in the second set of relevant images, a commonality metric value exclusive of the images in the first set of relevant images.

18.

Assignees

• Apple Inc

Inventors

• Schulze Anita Nariani
• Toft Rolf
• Hubel Paul M
• Tico Marius
• Zhou Jianping
• Brunner Ralph
• Molgaard Claus

Classifications

• H04N23/6845 Primary

  by combination of a plurality of images sequentially taken · CPC title

• H04N5/23277 Primary

  Electricity · mapped topic