Multi-frame super-resolution of image sequence with arbitrary motion patterns

Patent metadata

Field	Value
Publication number	US-9230303-B2
Application number	US-201414253149-A
Country	US
Kind code	B2
Filing date	Apr 15, 2014
Priority date	Apr 16, 2013
Publication date	Jan 5, 2016
Grant date	Jan 5, 2016

Abstract

A computer implemented method is provided for computing a two-way optical flow between a reference frame and one or more associated frames in an optical flow module. A forward warping operator and a backward warping operator can be generated between the reference frame and each of the one or more associated frames in a warping module. The forward warping operator and the backward warping operator provide motion compensation between the frames. Weights for each of the forward warping operators and the backward warping operators can be computed in a weight module. The weights correspond to uncertainty of motion estimation. A super resolution iteration algorithm can be calculated in a super-resolution iteration module.

First claim

The invention claimed is:

1. A computer implemented method, comprising the steps of:
computing, via one or more processors, a two-way optical flow between a reference frame and one or more associated frames;
generating, via said or another one or more processors, a forward warping operator and a backward warping operator between the reference frame and each of the one or more associated frames in a warping module;
computing, via said or another one or more processors, uncertainty of motion estimation weights for each of the forward warping operators and the backward warping operators in a weight module; and
calculating, via said or another one or more processors, a super resolution iteration algorithm in a super-resolution iteration module,

wherein the super resolution iteration algorithm is

I
H

(

k
+
1

)

=

I
H

(
k
)

+

Δ
⁢

⁢

t
⁡

(

∑

n
=
1

N

⁢

U

n
->
ref

⁢

W

n
->
ref

⁢

BD
T

⁢

U

ref
->
n

⁢

(

I
L
n

-

DBW

ref
->
n

⁢

I
H

(
k
)

)

+

λ
⁢

∂

R
⁡

(

∇

I
H

)

∂

I
H

)

,

 and
wherein I L n , n =1. . . N is a set of collected low resolution images I H is a reconstructed high resolution image, D is a down sampling operator, D T is an upsampling operator, B is a blurring matrix, W is a warping matrix that describes scene motion, U is an uncertainty of motion estimation, ref is a reference frame and n is a warped frame such that indexes n → ref and ref → n denote an operation towards the reference frame and vice versa, Δt is iteration time step, λ is a regularization multiplier, R(∇I H ) is a regularization term, and k is the iteration step.

2. The method of claim 1 , wherein the step of generating the forward warping operator and the backward warping operator is computed explicitly forward and backward in time.

3. The method of claim 1 , wherein the forward warping operator and the backward warping operator provide motion compensation between the frames.

4. A computer implemented system, comprising:
an optical flow module configured to compute a two-way optical flow between a reference frame and one or more associated fra

Assignees

• Kanaev Andrey V
• Miller Christopher W
• Us Navy

Inventors

• Kanaev Andrey V
• Miller Christopher W

Classifications

• G06K9/32

  Physics · mapped topic

• G06T7/2066

  Physics · mapped topic

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T7/269

  using gradient-based methods · CPC title