System and method for vision-based flight self-stabilization by deep gated recurrent Q-networks

Patent metadata

Field	Value
Publication number	US-10241520-B2
Application number	US-201615388662-A
Country	US
Kind code	B2
Filing date	Dec 22, 2016
Priority date	Dec 22, 2016
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

A system and a method for vision-based self-stabilization by deep gated recurrent Q-networks (DGRQNs) for unmanned arial vehicles (UAVs) are provided. The method comprises receiving a plurality of raw images captured by a camera installed on a UAV; receiving an initial reference image for stabilization and obtaining an initial camera pose from the initial reference image; extracting a fundamental matrix between consecutive images and estimating a current camera pose relative to the initial camera pose, wherein the camera pose includes an orientation and a location of the camera; based on the estimated current camera pose, predicting an action to counteract a lateral disturbance of the UAV based on the DGRQNs; and based on the predicted action to counteract the lateral disturbance of the UAV, driving the UAV back to the initial camera pose.

First claim

What is claimed is:

1. A method for vision-based self-stabilization by deep gated recurrent Q-networks (DGRQNs) for unmanned aerial vehicles (UAVs), comprising:
receiving a plurality of raw images captured by a camera installed on a UAV;
receiving an initial reference image for stabilization and obtaining an initial camera pose from the initial reference image;
extracting a fundamental matrix between consecutive images and estimating a current camera pose relative to the initial camera pose, wherein the camera pose includes an orientation and a location of the camera;
based on the estimated current camera pose, predicting an action to counteract a lateral disturbance of the UAV based on the DGRQNs, comprising:
encoding the estimated camera pose into a feature vector;
processing the feature vector by deep neuron networks (DNNs);
processing the feature vector by deep gated recurrent neural networks (DGRNs); and
processing the feature vector by Q-learning to obtain a difference between the estimated current camera pose and the initial camera pose; and

based on the predicted action to counteract the lateral disturbance of the UAV, driving the UAV back to the initial camera pose.

2. The method for vision-based self-stabilization by DGRQNs for UAVs according to claim 1 , wherein extracting a fundamental matrix between consecutive images and estimating a current camera pose relative to the initial camera pose further includes:
extracting the fundamental matrix and estimating the current camera pose relative to the initial camera pose from more than two consecutive images, comprising:
for each pair of consecutive images, finding a set of point correspondences using point trackers;
estimating a relative camera pose of a current image received at a current timestamp, wherein the relative camera pose of the current view includes a camera orientation and location of the current image with respect to a previous image received at a previous timestamp;
transforming the relative camera pose of the current image into a coordinate system of the initial reference image of the sequence;
storing the current image attributes including the camera pose and image points;
storing inlier matches between the previous image and the current image;
finding the point tracks across all the received images, given the initial reference image;
applying triangulation with multiple views to compute initial 3-D locations corresponding to the point tracks;
applying a bundle adjustment to refine the camera pose and 3-D points simultaneously; and
given all the received images, deriving a rotation matrix R and a translation matrix T relative to the initial reference image by matrix multiplications, wherein the rotation matrix R and translation matrix Tat a timestamp t is stored as H t =

H
t

=

[

R

T

0

1

]

.

3. The method for vision-based self-stabilization by DGRQNs for UAVs according to claim 1 , wherein extracting a fundamental matrix between consecutive images and estimating a current camera pose relative to the initial camera pose further includes:
extracting the fundamental matrix between two consecutive images and estimating the current camera pose relative to the initial camera pose from two consecutive images, wherein the fundamental matrix F is calculated as F=K 2 −T [T] x RK 1 −1 , where [T] x denotes a skew symmetric matrix and expressed as

[
T
]

X

=

[

0

-

t
3

t
2

t
3

0

-

t
1

-

t
2

t
1

0

]

,

K 1 and K 2 respectively camera matrices for a first image and a second image of the two consecutive images, R denotes a rotation matrix, and T denotes a translation matrix; and
the current camera pose is calculated as R 1 =UWV T , R 2 =UW T V T , T 1 =U 3 , T 2 =−U 3 , where

W
=

[

0

-
1

0

1

0

0

0

0

1

]

,

E=[T] x R=UΣV T , E denotes an essential matrix, R 1 is valid when det(R 1 )=1, R 2 is valid when det(R 2 )=1, T 1 is valid when z value of a 3D point is positive, and T 2 is valid when z value of a 3D point is positive, and the rotation matrix R and translation

Assignees

• Tcl Res America Inc

Inventors

• Sun Xinyao
• Liao Xinpeng
• Ren Xiaobo
• Wang Haohong

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title