Unified bracketing approach for imaging

Patent metadata

Field	Value
Publication number	US-10977778-B2
Application number	US-201916426448-A
Country	US
Kind code	B2
Filing date	May 30, 2019
Priority date	Jun 1, 2018
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

Devices, methods, and computer-readable media are disclosed describing an adaptive approach for image bracket selection and fusion, e.g., to generate low noise and high dynamic range (HDR) images in a wide variety of capturing conditions. An incoming image stream may be obtained from an image capture device, wherein the incoming image stream comprises a variety of differently-exposed captures, e.g., EV0 images, EV− images, EV+ images, long exposure (or synthetic long exposure) images, EV0/EV− image pairs, etc., which are received according to a particular pattern. When a capture request is received, a set of rules and/or a decision tree may be used to evaluate one or more capture conditions associated with the images from the incoming image stream and determine which two or more images to select for a fusion operation. A noise reduction process may optionally be performed on the selected images before (or after) the registration and fusion operations.

First claim

What is claimed is:

1. A device, comprising:
a memory;
one or more image capture devices;
a display;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain an incoming image stream from the one or more image capture devices;
obtain one or more capture conditions associated with the capture of the incoming image stream;
receive a capture request via the user interface;
select, in response to the capture request, two or more images from the incoming image stream based, at least in part, on the one or more capture conditions, wherein at least one of the selected images from the incoming image stream comprises an image captured before the receiving of the capture request;
register the selected two or more images;
fuse the registered two or more images to form a fused image; and
store the fused image in the memory.

2. The device of claim 1 , wherein the incoming image stream comprises images with two or more different exposure values.

3. The device of claim 2 , wherein the incoming image stream comprises a plurality of sequentially captured images, wherein the exposure values of the plurality of sequentially captured images follows a pattern of exposure values.

4. The device of claim 1 , wherein at least one of the selected images from the incoming image stream comprises an image captured after the receiving of the capture request.

5. The device of claim 4 , wherein the image captured after the receiving of the capture request comprises a long exposure image.

6. The device of claim 5 , wherein an exposure time of the long exposure image is based, at least in part, on ambient light levels around the one or more image capture devices.

7. The device of claim 1 , wherein the one or more capture conditions comprises at least one of the following:
ambient light levels around the one or more image capture devices;
whether an Optical Image Stabilization (OIS) mode of the one or more image capture devices has been engaged;
whether an High Dynamic Range (HDR) mode of the one or more image capture devices has been engaged;
whether flicker has been detected in one or more of the images from the incoming image stream;
whether it is permissible to disrupt the incoming image stream with a long exposure image capture;
whether the incoming image stream comprises image frame pairs with differing exposure values; and
positional information of the one or more image capture devices during the capture of the incoming image stream.

8. The device of claim 1 , wherein the selected two or more images comprise: at least two EV0 images.

9. The device of claim 8 , wherein the selected two or more images further comprise: at least one EV− image.

10. The device of claim 9 , wherein the selected two or more images further comprise: at least one long exposure image.

11. The device of claim 1 , wherein the selected two or more images from the incoming image stream further comprise: two images captured before the receiving of the capture request, wherein the two images are captured consecutively, and wherein the two images are captured with different exposure values.

12. The device of claim 11 , wherein one of the two images is selected to serve as the reference image for the registration of the selected two or more images.

13. A non-transitory computer-readable medium comprising instructions stored thereon to cause one or more processors to:
obtain an incoming image stream from one or more image capture devices;
obtain one or more capture conditions associated with the capture of the incoming image stream;
receive a capture request from at least one of the one or more image capture devices;
select, in response to the capture request, two or more images from the incoming image stream based, at least in part, on the one or more capture conditions, wherein at least one of the selected images from the incoming image stream comprises an image captured before the receiving of the capture request;
register the selected two or more images;
fuse the registered two or more images to form a fused image; and
store the fused image in the memory.

14. The non-transitory computer-readable medium of claim 13 , wherein the selected two or more images further comprise: at least one EV− image.

15. The non-transitory computer-readable medium of claim 14 , wherein the selected two or more images further comprise: at least one long exposure image.

16. The non-transitory computer-readable medium of claim 13 , wherein the selected two or more images from the incoming image stream further comprise: two images captured before the receiving of the capture request, wherein the two images are captured consecutively, and wherein the two images are captured with different exposure values.

17. The non-transitory computer-readable medium of claim 16 , wherein one of the two images is selected to serve as the reference image for the registration of the selected two or more images.

18. A method, comprising:
obtaining an incoming image stream from one or more image capture devices;
obtaining one or more capture conditions associated with the capture of the incoming image stream;
receiving a capture request from at least one of the one or more image capture devices;
selecting, in response to the capture request, two or more images from the incoming image stream based, at least in part, on the one or more capture conditions, wherein at least one of the selected images from the incoming image stream comprises an image captured before the receiving of the capture request;
registering the selected two or more images;
fusing the registered two or more images to form a fused image; and
storing the fused image in a memory.

19. The method of claim 18 , further comprising: performing noise reduction on the fused image.

20. The method of claim 18 , wherein the selected two or more images from the incoming image stream further comprise: two images captured before the receiving of the capture request, wherein the two images are captured consecutively, and wherein the two images are captured with different exposure values.

Assignees

• Apple Inc

Inventors

• Tico Marius
• Deshpande Alok
• Nariani Schulze Anita
• Sun Hao
• Hu Jun
• Faryabi-Allen Justin Michael
• Poulsen Morten
• Wong Tak Shing
• Cheng Wu

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N23/687

  by shifting the lens or sensor position · CPC title

• H04N23/741 Primary

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title