Robust image motion detection using scene analysis and image frame pairs

Patent metadata

Field	Value
Publication number	US-11113801-B1
Application number	US-201916563327-A
Country	US
Kind code	B1
Filing date	Sep 6, 2019
Priority date	Sep 11, 2018
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

Devices, methods, and computer-readable media describing an adaptive approach to reference image selection are disclosed herein, e.g., to generate fused images with reduced motion distortion. More particularly, an incoming image stream may be obtained from an image capture device, which image stream may comprise a variety of different image captures, e.g., including “image frame pairs” (IFPs) that are captured consecutively, wherein the images in a given IFP are captured with differing exposure settings. When a capture request is received at the image capture device, the image capture device may select two or more images from the incoming image stream for fusion, e.g., including at least one IFP. In some embodiments, one of the images from the at least one IFP will be designated as the reference image for a fusion operation, e.g., based on a robust motion detection analysis process performed on the images of the at least one IFP.

First claim

What is claimed is:

1. A device, comprising:
a memory;
one or more image capture devices;
a display;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain an incoming image stream from the one or more image capture devices;
select two or more images from the incoming image stream, wherein the selected two or more images comprise at least one image frame pair (IFP), wherein the IFP comprises a first image frame and a second image frame that were captured consecutively and with differing exposure settings;
determine one of the first image frame or the second image frame to be designated as a reference image, wherein the determination is based, at least in part, on a comparison of an amount of motion in the first image frame to an amount of motion in the second image frame;
determine one or more of the selected two or more images not designated as a reference image to be designated as candidate images;
register the one or more candidate images with the reference image;
fuse the registered one or more candidate images and the reference image, resulting in a fused image; and
store the fused image in the memory.

2. The device of claim 1 , wherein the second image frame comprises an image that is underexposed relative to the first image frame.

3. The device of claim 1 , wherein the first image frame comprises an EV0 image.

4. The device of claim 2 , wherein the second image frame comprises an EV− image.

5. The device of claim 1 , wherein the incoming image stream comprises a plurality of IFPs, and wherein the instructions to select two or more images comprising the at least one IFP further comprise instructions to select the at least one IFP from among the plurality of IFPs based on an evaluation of at least one criteria.

6. The device of claim 1 , wherein the comparison of an amount of motion in the first image frame to an amount of motion in the second image frame further comprises a comparison of an amount of motion within a Region of Interest (ROI) in the first image frame and second image frame.

7. The device of claim 6 , wherein the comparison of an amount of motion in the first image frame to an amount of motion in the second image frame further comprises masking out at least a portion of the ROI from the comparison.

8. The device of claim 6 , wherein the ROI comprises a plurality of non-overlapping image portions overlaying the first image frame and second image frame.

9. The device of claim 8 , wherein the plurality of non-overlapping image portions are non-contiguous.

10. The device of claim 6 , wherein a location or size of the ROI is based, at least in part on one or more of the following: an orientation of the image capture device during capture of the first image frame or second image frame; a portion of the first image frame or second image frame to which the image capture device's Auto Focus (AF) and/or Auto Exposure (AE) mechanisms have been directed; user input; and high-level content analysis of the first image frame or second image frame.

11. The device of claim 2 , wherein the comparison of an amount of motion in the first image frame to an amount of motion in the second image frame further comprises:
computing a first gradient map over at least a portion of the first image frame;
computing a second gradient map over a portion of the second image frame corresponding to the at least a portion of the first image frame; and
comparing the first gradient map to the second gradient map to determine an amount of difference.

12. The device of claim 11 , wherein the instructions to determine one of the first image frame or the second image frame to be designated as a reference image further comprise instructions to:
determine the first image frame to be designated as the reference image if the determined amount of difference does not exceed a motion threshold; and
determine the second image frame to be designated as the reference image if the determined amount of difference exceeds the motion threshold.

13. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain an incoming image stream from one or more image capture devices;
select two or more images from the incoming image stream, wherein the selected two or more images comprise at least one image frame pair (IFP), wherein the IFP comprises a first image frame and a second image frame that were captured consecutively and with differing exposure settings;
determine one of the first image frame or the second image frame to be designated as a reference image, wherein the determination is based, at least in part, on a comparison of an amount of motion in the first image frame to an amount of motion in the second image frame;
determine one or more of the selected two or more images not designated as a reference image to be designated as candidate images;
register the one or more candidate images with the reference image;
fuse the registered one or more candidate images and the reference image, resulting in a fused image; and
store the fused image in a memory.

14. The non-transitory program storage device of claim 13 , wherein the second image frame comprises an image that is underexposed relative to the first image frame.

15. The non-transitory program storage of claim 14 , wherein the comparison of an amount of motion in the first image frame to an amount of motion in the second image frame further comprises:
computing a first gradient map over at least a portion of the first image frame;
computing a second gradient map over a portion of the second image frame corresponding to the at least a portion of the first image frame; and
comparing the first gradient map to the second gradient map to determine an amount of difference.

16. The non-transitory program storage device of claim 15 , wherein the instructions to determine one of the first image frame or the second image frame to be designated as a reference image further comprise instructions to:
determine the first image frame to be designated as the reference image if the determined amount of difference does not exceed a motion threshold; and
determine the second image frame to be designated as the reference image if the determined amount of difference exceeds the motion threshold.

17. An image processing method, comprising:
obtaining an incoming image stream from one or more image capture devices;
selecting two or more images from the incoming image stream, wherein the selected two or more images comprise at least one image frame pair (IFP), wherein the IFP comprises a first image frame and a second image frame that were captured consecutively and with differing exposure settings;
determining one of the first image frame or the second image frame to be designated as a reference image, wherein the determination is based, at least in part, on a comparison of an amount of motion in the first image frame to an amount of motion in the second image frame;
determining one or more of the selected two or more images not designated as a reference image to be designated as candidate images;
registering the one or more candidate images with the reference image;
fusing the registered one or more candidate images and the reference image, resulting in a fused image; and
storing the fused image in a memory.

18. The method of claim 17 , wherein the second image frame comprises an image that is underexposed relat

Assignees

• Apple Inc

Inventors

• Cheng Wu

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/10144

  Varying exposure · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title