Fast robust method for compensating ego-translations and detecting independent moving objects in video captured with a moving camera

Patent metadata

Field	Value
Publication number	US-10078902-B1
Application number	US-201615250665-A
Country	US
Kind code	B1
Filing date	Aug 29, 2016
Priority date	Aug 27, 2015
Publication date	Sep 18, 2018
Grant date	Sep 18, 2018

Abstract

Described is a system for compensating ego-translations in video captured with a moving camera. Translative ego-motion is estimated on a sequence of image frames captured by a moving camera by minimizing a cost function that is based on at least one image frame difference between consecutive image frames. An alternating one directional search is performed to minimize the cost function to find an optimal translation. The optimal translation is applied to the sequence of image frames, resulting in a sequence of image frames with ego-translations.

First claim

What is claimed is:

1. A system for compensating ego-translations in video captured with a moving camera, the system comprising:
a video camera, wherein the video camera comprises:
one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
estimating translative ego-motion on a sequence of image frames captured by a moving camera by minimizing a cost function that is based on at least one image frame difference, wherein I and I′ represent consecutive image frames in the sequence of image frames, and wherein h and v represent shift translations, wherein an optimal translation (h, v) to compensate camera ego-motion is determined according to the following:

min

h
,
v

⁢

∑

i
⁢

⁢
j

⁢



I
′

⁡

(

i
+
h

,

j
+
v

)

-

I
⁡

(

i
,
j

)



,

where i and j represent coordinates of a position in an image frame;
wherein an alternating one directional search is performed to minimize the cost function to find the optimal translations; and
applying the optimal translations to the sequence of image frames, resulting in a sequence of image frames with ego-translations.

2. The system as set forth in claim 1 , wherein the one or more processors further perform operations of:
initiating a search for the optimal translation from an initial point;
searching as far as possible in one of a horizontal or a vertical direction until reaching a point representing a local minimum for that direction;
starting at the point representing the local minimum, searching as far as possible in the other of the horizontal or vertical direction; and
alternating searching in the horizontal and vertical directions until a global minimum is reached, wherein the optimal translation is the global minimum.

3. The system as set forth in claim 2 , wherein (h 0 , v 0 ) is the initial point, and wherein the one or more processors perform operations of:
(a) finding h 1 such that
min h f(h,v 0 ) at h=h 1 ;

(b) finding v 1 such that
min v f(h 1 ,v) at v=v 1 ;

(c) let Δh=|h 1 −h 0 |, Δv=|v 1 −v 0 |, then resetting h 0 :=h 1 and v 0 :=v 1 , where:=denotes “is defined as”;
(d) repeating operations (a)-(c) until Δh and Δv are predetermined values.

4. The system as set forth in claim 2 , wherein the optimal translation found in a current image frame is used as the initial point for a next image frame in the sequence of image frames to increase a search speed in finding the optimal translation in the next image frame.

5. The system as set forth in claim 1 , wherein the one or more processors further perform an operation of detecting and highlighting independent moving objects in the sequence of image frames with ego-translations.

6. The system as set forth in claim 5 , wherein the independent moving objects are detected by multiplying two consecutive image frame differences d 1 and d 2 , wherein I 1 , I 2 , I 3 are three consecutive image frames in the sequence of image frames, and (h 1 , v 1 ) and (h 2 , v 2 ) are corresponding optimal translations to compensate the camera ego-motion, wherein
d 1( i,j )=| I 1( i,j )− I 2( i+h 1, j+v 1)| and

d 2( i,j )=| I 2( i,j )− I 3( i+h 2, j+v 2)|,

wherein pixels for independent-moving objects are determined according to the following:
m ( i,j )= d 1( i,j ) d 2( i+h 1, j+v 1),

where m is an image map of detected object pixels, and (i, j) is a position in the image map.

7. The system as set forth in claim 5 , wherein independent moving objects are detected by multiplying image frame differences from non-adjacent image frames.

8. A computer implemented method for compensating ego-translations in video captured with a moving camera, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
estimating translative ego-motion on a sequence of image frames captured by a moving camera by minimizing a cost function that is based on at least one image frame difference, wherein I and I′ represent consecutive image frames in the sequence of image frames, and wherein h and v represent shift translations, wherein an optimal translation (h, v) to compensate camera ego-motion is determined according to the following:

min

h
,
v

⁢

∑

i
⁢

⁢
j

⁢



I
′

⁡

(

i
+
h

,

j
+
v

)

-

I

Assignees

• Hrl Lab Llc

Inventors

• Cao Yongqiang
• Srinivasa Narayan

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/13 Primary

  Satellite images · CPC title

• G06K9/0063

  Physics · mapped topic

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title