Method and apparatus of image representation and processing for dynamic vision sensor

Patent metadata

Field	Value
Publication number	US-9934557-B2
Application number	US-201615148657-A
Country	US
Kind code	B2
Filing date	May 6, 2016
Priority date	Mar 22, 2016
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

An apparatus and a method. The apparatus includes an image representation unit configured to receive a sequence of frames generated from events sensed by a dynamic vision sensor (DVS) and generate a confidence map from non-noise events; and an image denoising unit connected to the image representation unit and configured to denoise an image in a spatio-temporal domain. The method includes receiving, by an image representation unit, a sequence of frames generated from events sensed by a DVS, and generating a confidence map from non-noise events; and denoising, by an image denoising unit connected to the image representation unit, images formed from the frames in a spatio-temporal domain.

First claim

What is claimed is:

1. An apparatus, comprising:
an image representation unit configured to receive a sequence of frames generated from events sensed by a dynamic vision sensor (DVS), determine N 1 , events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 discard an associated event as noise, determine a neighborhood density for each non-noise event, determine confidence events as events with neighborhood densities greater than threshold T 2 , and generate a confidence map from the confidence events, where N 1 , N 2 , T 1 , and T 2 are each integers; and
an image denoising unit connected to the image representation unit and configured to determine N 2 events within the neighborhood and the time W of each event, determine C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determine noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoise an image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers.

2. The apparatus of claim 1 , wherein each event is associated with four values representing an event state, wherein the four values include an x and a y coordinate of a pixel indicating a location of the event, either a value of +1 to indicate a positive change in luminance of the event, a value of −1 to indicate a negative change in luminance of the event, or a value of 0 to indicate no change in luminance of the event as compared to an immediately preceding event state of the associated location, and a timestamp of the event.

3. The apparatus of claim 1 , wherein the image representation unit is further configured to determine, for each event not discarded, a neighborhood density I(valid_evt) as follows;

I
⁡

(

valid
evt

)

=

{

M

max
⁡

(

R
1

,
S

)

,

when
⁢

⁢
M

≤

R
2

M
k

,

when
⁢

⁢
M

>

R
2

,

where M is a number of events within a predetermined neighborhood of the event, including the event, S is a predetermined time window, R 1 is a threshold for avoiding values of S that are less than R 1 , R 2 is a threshold when M is larger than a predetermined value, and k is a predetermined constant.

4. An apparatus, comprising:
a dynamic vision sensor (DVS) configured to generate a stream of events;
a sampling unit connected to the DVS and configured to sample the stream of events;
an image formation unit connected to the sampling unit and configured to form an image for each sample of the stream of events by determining N 1 events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 , discarding an associated event as noise, determining a neighborhood density for each non-noise event, and determining confidence events as events with neighborhood densities greater than threshold T 2 , where N 1 , N 2 , T 1 , and T 2 are each integers;
an image representation unit connected to the image formation unit and configured to generate a confidence map from confidence events;
an image undistortion unit connected to the image representation unit and configured to compensate for distortion in frames by determining N 2 events within the neighborhood and the time W of each event, determining C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determining noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoising an image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers; and
an image matching unit connected to the image undistortion unit and the sampling unit and configured to match frames and adjust a sampling method of the sampling unit, if necessary.

5. The apparatus of claim 4 , wherein the image matching unit is further configured to match frames using weighted frame-to-frame matching as follows:
arg min½Σ i C ( p i )∥ I (1) ( p i )− I (2) ( T×p i )∥ 2 ,

where arg min ƒ(x) is an argument-of-a-minimum function that determines a value x for which a function ƒ(x) attains its minimum, where i is a location in a confidence map C (1) of a first image I (1) , p i is a coordinate for location i, I (2) is a second image, and T is a transformation matrix that minimizes the arg min function.

6. A method, comprising:
receiving, by an image representation unit, a sequence of frames generated from events sensed by a dynamic vision sensor (DVS), determining N 1 events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 , discarding an associated event as noise, determining a neighborhood density for each non-noise event, and determining confidence events as events with neighborhood densities greater than threshold T 2 , and generating a confidence map from non-noise events, where N 1 , N 2 , T 1 , and T 2 are each integers; and
denoising, by an image denoising unit connected to the image representation unit, images formed from the frames in a spatio-temporal domain by determining N 2 events within the neighborhood and the time W of each event, determining C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determining noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoising the image in a spatio-temporal domain by discarding the noi

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Ji Zhengping
• Lee Kyoobin
• Zhang Qiang
• Wang Yibing Michelle
• Ryu Hyun Surk
• Ovsiannikov Ilia

Classifications

• H04N25/60

  Noise processing, e.g. detecting, correcting, reducing or removing noise · CPC title

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06K9/00

  Physics · mapped topic