Method and apparatus for motion estimation

Patent metadata

Field	Value
Publication number	US-10229504-B2
Application number	US-201414777384-A
Country	US
Kind code	B2
Filing date	Mar 14, 2014
Priority date	Mar 18, 2013
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

A method of estimating motion between a pair of image frames of a given scene comprises calculating respective integral images for each of the image frames and selecting at least one corresponding region of interest within each frame. For each region of interest, an integral image profile from each integral image is calculated, each profile comprising an array of elements, each element comprising a sum of pixel intensities from successive swaths of the region of interest for the frame. Integral image profiles are correlated to determine a relative displacement of the region of interest between the pair of frames. Each region of interest is divided into a plurality of further regions of interest before repeating until a required hierarchy of estimated motion for successively divided regions of interest is provided.

First claim

The invention claimed is:

1. A method of estimating motion between a pair of image frames of a given scene comprising the steps of:
a) calculating respective integral images for each of said image frames;
b) selecting at least one corresponding region of interest within each frame; and
c) for each region of interest:
i. calculating an integral image profile from each integral image, each profile comprising an array of elements, each element comprising a sum of pixel intensities from successive swaths of said region of interest for said frame;
ii. correlating said integral image profiles to determine a relative displacement of said region of interest between said pair of frames; and
iii. dividing each region of interest into a plurality of further regions of interest; and

d) repeating step c) until a required hierarchy of estimated motion for successively divided regions of interest is provided, wherein said calculating an integral image profile comprises sub-sampling said integral image at a first sub-sampling interval at a first selected level of said required hierarchy and for each repetition of step c), sub-sampling said integral image at successively smaller sub-sampling intervals.

2. A method according to claim 1 , wherein step b) comprises initially selecting a complete image frame as a region of interest.

3. A method according to claim 1 , wherein said dividing comprises dividing each region horizontally and vertically into an integer number of regions of interest.

4. A method according to claim 3 , wherein said integer number is 2.

5. A method according to claim 1 , further comprising: designating a first of said image frames as a reference frame and a second of said image frames as a target frame, displacing said divided regions of said target frame at one level of said required hierarchy according to a relative displacement determined for a region of interest corresponding to said divided regions of interest at a higher level of the required hierarchy.

6. A method according to claim 5 , wherein said displacing is responsive to said correlating determining more than one displacement within said region of interest for displacing said divided regions of interest within said target image differently according to said more than one displacement.

7. A method according to claim 5 , wherein said dividing is responsive to said correlating determining more than one displacement within said region of interest for determining a non-uniform division of said region of interest and for dividing said region of interest differently according to said non-uniform division.

8. A method according to claim 5 , further comprising calculating a full size integral image for a target frame and calculating a reduced size integral image for a reference frame, said reduced size integral image comprising only integral image data for pixels of said reference frame corresponding to boundaries of regions of interest within said required hierarchy.

9. A method according to claim 8 , further comprising selecting said reference and target frames from a sequence of image frames so that said reference frame alternately precedes and succeeds said target frame.

10. A method according to claim 5 , further comprising receiving from a camera motion sensor, a measure of a camera movement between said pair of image frames; and wherein said selecting comprises selecting a plurality of regions within said image, at least one region of interest within said target frame being displaced according to said measure of camera movement.

11. A method according to claim 1 , further comprising sampling each pixel of said integral image at a bottom level of said required hierarchy.

12. A method according to claim 11 , further comprising interpolating said integral image profiles for each of said image frames at bottom level of said required hierarchy to provide a sub-pixel measure of displacement for regions of interest at the bottom level of said required hierarchy.

13. A method according to claim 1 , further comprising generating a geometrical transformation indicating a relative movement of said pair of frames, based on an array of motion vectors determined for regions of interest at a bottom level of said required hierarchy.

14. A method according to claim 13 , further comprising filtering said motion vectors prior to said generating to eliminate outlying motion vectors.

15. A method according to claim 1 , wherein, for each repetition of step c), said sub-sampling intervals are halved.

16. A method according to claim 1 , wherein said relative displacement comprises one or both of vertical displacement or horizontal displacement.

17. A method of estimating motion between a pair of image frames of a given scene comprising:
a) receiving from a camera motion sensor, a measure of camera movement between said pair of image frames;
b) for a first of said image frames designated as a reference frame and for a second of said image frames designated as a target frame, calculating an integral image profile based on integral images of the reference and target frames;
c) selecting a plurality of regions of interest within each frame;
d) displacing at least one of said regions of interest within said target frame according to said measure of camera movement;
e) for each region of interest: i. based on the integral image profiles of the reference and target frames, correlating image information for said region of interest to determine a relative displacement of said region of interest between said pair of frames; and ii. dividing said region of interest into a plurality of further regions of interest; and
f) repeating step e) until a required hierarchy of estimated motion for successively divided regions of interest is provided, wherein said calculating an integral image profile comprises sub-sampling said integral images at a first sub-sampling interval at a first selected level of said required hierarchy and for each repetition of step e), sub-sampling said integral images at successively smaller sub-sampling intervals.

18. An image processing device arranged to acquire a sequence of image frames of a scene and to estimate motion between a pair of image frames of said scene, comprising:
an integral image generator that:
calculates a first integral image for a first image frame; and
calculates a second integral image for a second image frame;

a registration engine that performs the steps of:
selecting a plurality of corresponding regions of interest within the first image frame and the second image frame; and
for each region of interest of the plurality of corresponding regions of interest:
calculating a first integral image profile based on the first integral image and a second integral image profile based on the second integral image;
correlating the first integral image profile with the second integral image profile to determine a relative displacement of the region of interest between the first image frame and the second image frame;
dividing each region of interest into a plurality of further regions of interest; and
repeating the calculating, correlating, and dividing steps until a hierarchy of estimated motion for successively divided regions of interest is provided, wherein the registration engine calculates each integral image profile by sub-sampling said integral images at a first sub-sampling interval at a first selected level of said hierarchy and, for each repetition of dividing, sub-sampling said integral images at successively smaller sub-sampling intervals.

Assignees

• Fotonation Ltd

Inventors

• Bigioi Petronel
• Corcoran Peter
• Stec Piotr

Classifications

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N19/53 Primary

  Multi-resolution motion estimation; Hierarchical motion estimation · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title