System and method for enhancing target tracking via detector and tracker fusion for unmanned aerial vehicles

Patent metadata

Field	Value
Publication number	US-10140719-B2
Application number	US-201615387846-A
Country	US
Kind code	B2
Filing date	Dec 22, 2016
Priority date	Dec 22, 2016
Publication date	Nov 27, 2018
Grant date	Nov 27, 2018

Abstract

A system and a method for enhancing target tracking via detector and tracker fusion for unmanned aerial vehicles (UAVs) are provided. The method comprises receiving at least one raw input image of objects to be detected; based on the at least one raw input image of objects, generating the objects' candidate information; based on the objects' candidate information, calculating location and velocity estimation of an object at a current timestamp based on a detector and tracker fusion; and based on the location and velocity estimation of the object at the current timestamp, predicting the location and velocity estimation of the object at a future timestamp.

First claim

What is claimed is:

1. A method for enhancing target tracking via detector and tracker fusion for Unmanned Aerial Vehicles (UAVs), comprising:
receiving at least one raw input image of objects to be detected;
based on the at least one raw input image of objects, generating the objects' candidate information;
based on the objects' candidate information, calculating location and velocity estimation of an object at a current timestamp based on a detector and tracker fusion, comprising:
generating a binary feature canvas;
processing the binary feature canvas based on an attention mechanism to focus on relevant parts of the binary feature canvas;
extracting features of the objects from the processed binary feature canvas; and
calculating the location and velocity estimation of the object at the current timestamp;
wherein processing the binary feature canvas based on the attention mechanism to focus on relevant parts of the binary feature canvas further including:
generating an attention mask M t (e t-1 ) and applying the attention mask M t (e t-1 ) to each channel of the feature canvas c t , respectively; and
obtaining a masked feature canvas,
wherein the attention mask M t (e t-1 ) is generated by a mixture of N×N Gaussians, each Gaussian (i,j) has a center at

(

rx

t
-
1

+

lx

t
-
1

2

+

vx

t
-
1

+

(

i
-

N
2

-
0.5

)

,

⁢

ry

t
-
1

+

ly

t
-
1

2

+

vy

t
-
1

+

(

i
-

N
2

)

)

⁢

⁢
and
⁢

⁢
σ

=

S
frame

S

Pt
-
1

,

 where S frame denotes an area of the raw input image, and S p t-1 denotes an area of the object's estimated bounding box at a timestamp (t−1), and
m(c t , e t-1 )=M t (e t-1 )·c t , where m(c t , e t-1 ) denotes the masked feature canvas; and

based on the location and velocity estimation of the object at the current timestamp, predicting the location and velocity estimation of the object at a future timestamp.

2. The method for enhancing target tracking via detector and tracker fusion for UAVs according to claim 1 , wherein receiving at least one raw input image of objects to be detected further including:
receiving the raw input image of objects to be detected through a single lens camera mounted on the UAV.

3. The method for enhancing target tracking via detector and tracker fusion for UAVs according to claim 1 , wherein based on the at least one raw input image of objects, generating the objects' candidate information further including:
based on a pre-trained histogram-of-oriented-gradient (HOG) algorithm, generating a bounding box bb (lx,ly,rx,ry) corresponding to the object at the current timestamp, wherein the bounding box information includes location of left upper corner (lx,ly,) and location of bottom-right corner (rx,ry);
based on Kanade-Lucas-Tomasi (KLT) approach, generating the object's location information at each timestamp according to trackers feature points P KLT ={p 1 (x 1 ,y 1 ) , p 2 (x 2 ,y 2 ) . . . p n (x n ,y n ) }; and
based on learnt discriminative correlation filters on scaled pyramid representations for both translation and scale estimation, generating a bounding box bb corr .

4. The method for enhancing target tracking via detector and tracker fusion for UAVs according to claim 1 , wherein:
the binary feature canvas is denoted by c t , c t represents vision information in a feature space at a current timestamp t;
the binary feature canvas has a size (h, w) which is the same as a size of the raw input image; and
the binary feature canvas has three channels corresponding to information of {bb,P KLT ,bb corr } with binary values using the one-hot encoding, wherein when the channel represents the bounding box (bb, bb corr ,), then a pixel inside the bounding box is set to be 1 and a pixel outside the bounding box to be 0, and when the channel represents the trackers feature points (P KLT ), then positions of the trackers feature points on the binary feature canvas is set to be 1 and positions of the trackers feature points beyond the binary feature canvas to be 0.

Assignees

• Tcl Res America Inc

Inventors

• Sun Xinyao
• Liao Xinpeng
• Ren Xiaobo
• Wang Haohong

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title