Image set alignment and combination processing

Patent metadata

Field	Value
Publication number	US-9531962-B2
Application number	US-201514682895-A
Country	US
Kind code	B2
Filing date	Apr 9, 2015
Priority date	Jul 24, 2014
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

A method includes identifying feature points in an image in images generated of a scene by a camera and identifying locations of the identified feature points in remaining images in the images. The method also includes selecting a group of the identified feature points indicative of relative motion of the camera between image captures and aligning a set of the images using the selected group of feature points. The method may further include selecting a reference image from the set of aligned images, weighting other images the set, and combining the reference image with the weighted images. Weighting of the other images may include, for each other image in the set, comparing the other image and the reference image to identify one or more moving objects in the other image and applying a weight to pixel locations in the other image.

First claim

What is claimed is:

1. A method for aligning and combining images, the method comprising:
identifying a plurality of feature points in an image in a plurality of images generated of a scene by a camera;
identifying locations of the identified feature points in remaining images in the plurality of images;
selecting a group of the identified feature points indicative of relative motion of the camera between image captures; and
aligning a set of the plurality of images using the selected group of feature points;
selecting a reference image from the set of aligned images;
weighting other images the set by, for each other image in the set:
comparing the other image and the reference image to identify one or more moving objects in the other image relative to the reference image; and
applying a weight to a plurality of pixel locations in the other image, a value of the weight for a pixel location decreasing based on an amount of relative movement of an object formed by the pixel location; and

combining the reference image with the weighted images.

2. A method for aligning images, the method comprising:
identifying a plurality of feature points in a reference image in a set of images generated of a scene by a camera;
identifying locations of the identified feature points in remaining images in the set of images;
selecting a group of the identified feature points indicative of relative motion of the camera between image captures; and
aligning at least one of the remaining images with the reference image using the selected group of feature points.

3. The method of claim 2 , wherein aligning the at least one remaining image comprises:
estimating a global affine transformation between the at least one remaining image and the reference image using locations of the selected group of feature points in the at least one remaining image and the reference image, and
aligning the at least one remaining image with the reference image based on the estimated global affine transformation.

4. The method of claim 3 , wherein aligning the at least one remaining image with the reference image further comprises warping one or more of the images by at least one of translating, scaling, and rotating to align the one or more images with the reference image.

5. The method of claim 2 , wherein selecting the group of the identified feature points and aligning the at least one remaining image with the reference image comprises:
segmenting the reference image and the at least one remaining image into a plurality of regions;
for each of the regions, estimating a local affine transformation between locations of feature points in each of the images in the set;
identifying movement of objects among the images based on estimation error of the local affine transformation for each of the regions; and
using only feature points in the regions having a local affine transformation estimation error less than a threshold to estimate a global affine transformation across the set of images.

6. The method of claim 5 , wherein selecting the group of the identified feature points and aligning the at least one remaining image with the reference image further comprises:
identifying a local affine transformation estimation error of each of the images in the set; and
using only feature points in the images having a local affine transformation estimation error less than a threshold to estimate the global affine transformation across the set of images.

7. The method of claim 2 , wherein identifying the plurality of feature points in the reference image comprises identifying the points having a gradient exceeding a threshold in more than one direction.

8. A method for combining images, the method comprising:
selecting a reference image from a set of images generated of a scene by a camera;
weighting other images the set by, for each other image in the set:
comparing the other image and the reference image to identify one or more moving objects in the other image relative to the reference image; and
applying a weight to a plurality of pixel locations in the other image, a value of the weight for a pixel location decreasing based on an amount of relative movement of an object formed by the pixel location; and

combining the reference image with the weighted images.

9. The method of claim 8 , wherein comparing the other image and the reference image and applying the weight comprises:
segmenting the reference image and the other image into a plurality of regions; and
for each of the regions:
comparing corresponding regions of the other image and the reference image; and
assigning a weight to a part or an entirety of the region based on similarities of the compared regions.

10. The method of claim 8 , wherein comparing the other image and the reference image and applying the weight comprises:
calculating a difference image between the other image and the reference image;
applying a convolution to the difference image;
assigning weight values for each pixel location in the difference image; and
applying a convolution to the weight values.

11. The method of claim 8 , wherein combining the reference image with the weighted images comprises removing noise in the reference image prior to combining the reference image with the weighted images.

12. The method of claim 8 , wherein combining the reference image with the weighted images comprises:
identifying locations of one or more moving objects in the reference image based on a motion threshold and weights applied to the pixel locations in the weighted images; and
removing noise from the identified locations of one or more moving objects in the reference image.

13. The method of claim 8 , wherein combining the reference image with the weighted images comprises:
identifying locations of one or more moving objects in the reference image based on a motion threshold and weights applied to the pixel locations in the weighted images; and
reducing image sharpening of the identified locations of one or more moving objects in the reference image.

14. An apparatus for aligning images, the apparatus comprising:
a memory configured to at least temporarily store a set of images generated of a scene by a camera; and
a controller configured to:
identify a plurality of feature points in a reference image in the set of images;
identify locations of the identified feature points in remaining images in the set of images;
select a group of the identified feature points indicative of relative motion of the camera between image captures; and
align at least one of the remaining images with the reference image using the selected group of feature points.

15. The apparatus of claim 14 , wherein to align the at least one remaining image the controller is configured to:
estimate a global affine transformation between the at least one remaining image and the reference image using locations of the selected group of feature points in the at least one remaining image and the reference image, and
align the at least one remaining image with the reference image based on the estimated global affine transformation.

16. The apparatus of claim 15 , wherein to align the at least one remaining image with the reference image the controller is further configured to warp one or more of the images by at least one of translating, scaling, and rotating to align the one or more images with the reference image.

17. The apparatus of claim 14 , wherein to select the group of the identified feature points and align the at least one remaining image with the reference image the

Assignees

• Samsung Electronics Co Ltd

Inventors

• Sezer Osman G
• Glotzbach John W
• Sheikh Hamid R

Classifications

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N5/2625 Primary

  for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect (sequence generated by event triggered capturing H04N7/188) · CPC title