Image registration methods for still image stabilization

Patent metadata

Field	Value
Publication number	US-9384552-B2
Application number	US-201313911793-A
Country	US
Kind code	B2
Filing date	Jun 6, 2013
Priority date	Jun 6, 2013
Publication date	Jul 5, 2016
Grant date	Jul 5, 2016

Abstract

Systems, methods, and computer readable media to improve image stabilization operations are described. A novel approach to pixel-based registration of non-reference images to a reference frame in a set of commonly captured images is disclosed which makes use of pyramid decomposition to more efficiently detect corners. The disclosed pixel-based registration operation may also be combined with motion sensor data-based registration approaches to register non-reference images with respect to the reference frame. When the registered non-reference images are combined with the pre-selected reference image, the resulting image is a quality stabilized image.

First claim

The invention claimed is:

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:
obtain a first image and a first one or more additional images;
obtain motion information corresponding to each of the first one or more additional images;
compensate for motion between the first image and each of the first one or more additional images based, at least in part, on each image's corresponding motion information to generate a second one or more additional images, by:
determining a transform matrix for each of a plurality of segments of each of the first one or more additional images based, at least in part, on the motion information, and
applying the transform matrix for each of the first one or more additional images to the corresponding first one or more additional image,

wherein each of the second one or more additional images has a corresponding image in the first one or more additional images, and wherein each segment corresponds to a plurality of pixel rows in an image; and
register the second one or more additional images with respect to the first image based, at least in part, on a pixel-level analysis of the first image and each of the second one or more additional images to generate an output image.

2. The non-transitory program storage device of claim 1 , further comprising instructions to cause the one or more programmable control devices to identify the first image as a reference image.

3. The non-transitory program storage device of claim 1 , wherein the motion information comprises motion information of each of the first one or more additional images with respect to the first image.

4. The non-transitory program storage device of claim 1 , wherein the instructions to cause the programmable control device to register the second one or more additional images based, at least in part, on a pixel-level analysis of the first image and one of the second one or more additional images comprise instructions to cause the programmable control device to:
apply pyramid decomposition to the first image and one of the second one or more additional images to generate a plurality of levels for each of the first image and the one of the second one or more additional images;
detect corners at each level of the first image and for each detected corner, wherein the instructions to cause the programmable control device to detect corners comprise instructions to cause the programmable control device to:
search for a corresponding corner in a corresponding level of the one of the second one or more additional images;
generate a transformation matrix based on locations of the corresponding corners found; and
apply the transformation matrix to the one of the second one or more additional images.

5. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:
obtain a first image and a first one or more additional images;
obtain motion information corresponding to each of the first one or more additional images;
determine a transform matrix for each of a plurality of segments of each of the first one or more additional images based, at least in part, on the motion information, wherein each segment corresponds to a plurality of pixel rows in an image;
apply the transform matrix for each of the first one or more additional images to the corresponding first one or more additional image; and
register the first one or more additional images with respect to the first image based, at least in part, on a pixel-level analysis of the first image, the first one or more additional images and the transform matrices for each of the first one or more additional images to generate an output image.

6. The non-transitory program storage device of claim 5 , wherein the motion information comprises gyroscope data.

7. The non-transitory program storage device of claim 5 , wherein the motion information comprises of motion of each of the first one or more additional images with respect to the first image.

8. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:
obtain a first image having a plurality of pixels arranged in rows;
obtain a first one or more additional images, wherein a plurality of pixels in the first image have corresponding pixels in each of the first one or more additional images;
obtain motion information corresponding to each of the first one or more additional images;
determine a first transform matrix for each segment of each of the first one or more additional images based, at least in part, on the motion information, wherein each segment corresponds to a one or more rows of pixels;
compensate for motion between the first image and each of the first one or more additional images based, at least in part, on each image's corresponding motion information to generate a second one or more additional images, wherein each of the second one or more additional images has a corresponding image in the first one or more additional images;
determine a second transform matrix for each of the first one or more additional images based, at least in part, on a pixel-level analysis of the first image and the second one or more additional images;
combine each image's first transform matrices and the corresponding second transform matrices to generate a transformation matrix for each segment of each of the first one or more additional images;
apply the transformation matrices to corresponding segments of the first one or more additional images to generated one or more transformed segments for each of the first one or more additional images; and
combine the transformed segments for each of the first one or more additional images to register the first one or more additional images.

9. The non-transitory program storage device of claim 8 , further comprising instructions to cause the one or more programmable control devices to identify the first image as a reference image.

10. The non-transitory program storage device of claim 8 , wherein the motion information comprises gyroscope data.

11. The non-transitory program storage device of claim 10 , wherein the motion information comprises motion information of each of the first one or more additional images with respect to the first image.

12. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:
obtain a first and a second images;
obtain motion information corresponding to the first and second images;
compensate for motion between the first and second images based, at least in part, on the motion information corresponding to the first and second images, wherein the instructions to compensate for motion between the first and second images further comprise instructions to:
apply pyramid decomposition on each of the first and the second images to generate a plurality of levels for each image;
detect corners at each level of the first image;
for each corner detected at each of level the first image, search for a corresponding corner in a corresponding level of the second image; and
generate a transformation matrix for each corresponding corner identified based on locations of the corresponding corners; and

apply the transformation matrices to the second image to register the second image, wherein the second image is registered based, at least in part, on a pixel-level analysis of the

Assignees

• Apple Inc

Inventors

• Zhou Jianping
• Hubel Paul
• Tico Marius
• Nariani Schulze Anita
• Toft Rolf

Classifications

• G06T7/0028 Primary

  Physics · mapped topic

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/33 Primary

  using feature-based methods · CPC title