Image alignment for burst mode images

Patent metadata

Field	Value
Publication number	US-2018158199-A1
Application number	US-201715676903-A
Country	US
Kind code	A1
Filing date	Aug 14, 2017
Priority date	Dec 6, 2016
Publication date	Jun 7, 2018
Grant date	—

Abstract

The present disclosure is directed towards systems and methods for generating a new aligned image from a plurality of burst image. The systems and methods subdivide a reference image into a plurality of local regions and a subsequent image into a plurality of corresponding local regions. Additionally, the systems and methods detect a plurality of feature points in each of the reference image and the subsequent image and determine matching feature point pairs between the reference image and the subsequent image. Based on the matching feature point pairs, the systems and methods determine at least one homography of the reference image to the subsequent image. Based on the homography, the systems and methods generate a new aligned image that is that is pixel-wise aligned to the reference image. Furthermore, the systems and methods refines boundaries between local regions of the new aligned image.

First claim

We claim:

1 . A computer-implemented method of generating aligned images from burst images, the method comprising:
detecting a plurality of feature points in a reference image and a subsequent image; determining a feature descriptor for each feature point of the plurality of feature points; based on the feature descriptor of each feature point, determining matching feature points between the reference image and the subsequent image; determining at least one homography that maps matching feature points between the reference image and the subsequent image; and warping the subsequent image based on the at least one homography to generate a new aligned image that is aligned to the reference image.

2 . The computer-implemented method of claim 1 , wherein determining a feature descriptor for each feature point of the plurality of feature points comprises analyzing the detected feature points and generating a descriptor vector for each detected feature point that encodes intensity information, pixel location information, and local neighborhood information regarding its respective feature point.

3 . The computer-implemented method of claim 1 , wherein determining matching feature points between the reference image and the subsequent image comprises:
predicting for a given feature point having a pixel location in the reference image a corresponding pixel location in the subsequent image; identifying a set of potentially matching feature points from the subsequent image; filtering, from the set of potentially matching feature points, any feature points having a descriptor distance above a threshold distance from the given feature point; and selecting a matching feature point from the set of potentially matching feature points having a highest similarity score as a match to the given feature point.

4 . The computer-implemented method of claim 3 , wherein identifying the set of potentially matching feature points from the subsequent image comprises comparing a feature descriptor of the given feature point with feature descriptors of feature points having a pixel location within a predetermined number of pixels of the predicted corresponding pixel location in the subsequent image.

5 . The computer-implemented method of claim 4 , wherein identifying the set of potentially matching feature points comprises:
sorting feature points within the predetermined number of pixels of the predicted corresponding pixel location according to the feature points' similarities with the given feature point; and filtering, from the sorted feature points, all but a predetermined number of most similar feature points.

6 . The computer-implemented method of claim 5 , wherein filtering, from the sorted feature points, all but the predetermined number of most similar feature points comprises filtering, from the sorted feature points, all but a three most similar feature points.

7 . The computer-implemented method of claim 5 , wherein sorting feature points within the predetermined number of pixels of the predicted corresponding pixel location according to the feature points' similarities with the given feature point comprises sorting the feature points according to a Hamming distance between the feature descriptors of the feature points and the feature descriptor of the given feature point.

8 . The computer-implemented method of claim 3 , determining matching feature points between the reference image and the subsequent image further comprises:
determining, from the set of potentially matching feature points, a median displacement vector from the pixel location of the given feature point; and filtering, from the set of potentially matching feature points, feature points having a displacement that deviates from the median displacement vector by more than a predetermined number of pixels.

9 . The computer-implemented method of claim 1 , wherein determining at least one homography that maps matching feature points between the reference image and the subsequent image comprises applying a homography motion model to each of a plurality of local regions individually to determine a homography for each local region that maps feature points in a local region of the reference image to feature points in a corresponding local region of the subsequent image.

10 . The computer-implemented method of claim 8 , wherein warping the subsequent image based on the at least one homography to generate a new aligned image that is aligned to the reference image comprises warping the subsequent image based on the homography of each local region.

11 . A system for generating aligned images from burst images, comprising:
a memory comprising a references image and a subsequent image; at least one processor communicatively coupled to the memory and storing instructions thereon, that, when executed by the at least one processor, cause the system to: subdivide the reference image into a plurality of local regions; subdivide the subsequent image into a plurality of corresponding local regions; detect a plurality of feature points in the local regions of the reference image; detect a plurality of feature points the corresponding local regions of the subsequent image; determine matching feature points between the reference image and the subsequent image; merge local regions of the reference image that do not include a threshold number of feature points with another local region of the reference image; based on the matching feature points between the reference image and the subsequent image, determine a homography for each local region and merged region of the reference image, the homography of each local region mapping feature points from a respective local region of the reference image to a respective corresponding local region of the subsequent image, and warp the subsequent image based on the homography of each local region and merged local region to generate a new aligned image that is aligned to the reference image.

12 . The system as recited in claim 11 , wherein the instructions when executed by the at least one processor cause the system to determine a homography for each local region of the reference image by performing steps comprising:
determining whether each local region of the plurality of local regions of the reference image includes at least a threshold number of feature points; if a given local region of the reference image includes at least the predetermined number of feature points, determining a homography of the given local region; and if a given local region of the reference image does not include at least the predetermined number of feature points, merging the given local region with an additional local region of the reference image to form a merged local region and determining the homography of the merged local region.

13 . The system as recited in claim 12 , wherein the instructions when executed by the at least one processor cause the system to determine the homography of the merged local region by performing steps comprising fitting a robust homography motion model to the merged local region.

14 . The system as recited in claim 12 , wherein the instructions when executed by the at least one processor cause the system to warp the subsequent image by interpolating homography flow vectors of the homographies that are proximate to borders shared by two or more of the corresponding local regions of the subsequent image.

15 . The system as recited in claim 11 , further comprising instructions that when executed by the at least one processor cause the system to:
determine whether corners of the new aligned image

Assignees

• Adobe Systems Inc

Inventors

• Wang Zhaowen
• Jin Hailin

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title