Method and device for processing DVS events

Patent metadata

Field	Value
Publication number	US-10445924-B2
Application number	US-201715811200-A
Country	US
Kind code	B2
Filing date	Nov 13, 2017
Priority date	Nov 14, 2016
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

Embodiments of the present invention provide a method and device for processing Dynamic Vision Sensor (DVS) events. The method comprises the operations of: acquiring a DVS event map sequence; for any DVS event map in the DVS event map sequence, extracting a DVS event feature descriptor, where the DVS event feature descriptor has a scale-invariant feature and/or a rotation-invariant feature; determining, according to the extracted DVS event feature descriptor, a three-dimensional space pose of the DVS event map at the current moment; and, generating, according to the three-dimensional space pose of each DVS event map, a DVS event map sequence with temporal consistency. The embodiments of the present invention are used for generating, from a DVS event map sequence, a DVS event map sequence with temporal consistency.

First claim

What is claimed is:

1. A method for processing Dynamic Vision Sensor (DVS) events, comprising:
acquiring a first DVS event map sequence comprising a first DVS event map, and a second DVS event map sequence comprising a second DVS event map, the first DVS event map sequence and the second DVS event map sequence being of a dual-camera DVS event map sequence;
extracting a first DVS event feature descriptor from the first DVS event map, and a second DVS event feature descriptor from the second DVS event map;
determining, based on the first DVS event feature descriptor, a three-dimensional space pose of the first DVS event map at a current moment; and
generating, based on the three-dimensional space pose of the first DVS event map, a generated DVS event map sequence with a temporal consistency for generating a video image,
wherein the first DVS event feature descriptor having at least one from among a scale-invariant feature and a rotation-invariant feature,
wherein the extracting the second DVS event feature descriptor comprises:
generating a vision pyramid for the second DVS event map, a number of layers of the vision pyramid being a first preset value, a down-sampling coefficient between any two adjacent layers of the vision pyramid being a second preset value; and

acquiring, based on the vision pyramid, a scale-invariant DVS event feature descriptor.

2. The method for processing Dynamic Vision Sensor (DVS) events according to claim 1 , further comprising, before the determining, based on the first DVS event feature descriptor, the three-dimensional space pose of the first DVS event map at the current moment:
determining, based on the first DVS event feature descriptor, a depth value corresponding to an event,
wherein the determining, based on the first DVS event feature descriptor, the three-dimensional space pose of the first DVS event map at the current moment comprises:
determining, based on the depth value corresponding to the event and the first DVS event feature descriptor, the three-dimensional space pose of the first DVS event map at the current moment.

3. The method for processing Dynamic Vision Sensor (DVS) events according to claim 2 , wherein the determining, based on the depth value corresponding to the event and the first DVS event feature descriptor, the three-dimensional space pose of the first DVS event map at the current moment comprises:
based on the depth value corresponding to the event and the first DVS event feature descriptor, matching the first DVS event map at the current moment with a key frame in the first DVS event map sequence; and
determining, based on a result of matching, the three-dimensional space pose of the first DVS event map at the current moment.

4. The method for processing Dynamic Vision Sensor (DVS) events according to claim 3 , further comprising, after the determining, based on the result of the matching, the three-dimensional space pose of the first DVS event map at the current moment:
adjusting the three-dimensional space pose of the first DVS event map at the current moment so that an accumulative error generated for the first DVS event map during the matching is less than a preset value,
wherein the generating, based on the three-dimensional space pose of the first DVS event map, the generated DVS event map sequence with the temporal consistency comprises:
generating, based on the three-dimensional space pose that has been adjusted, of the first DVS event map, the generated DVS event map sequence with the temporal consistency.

5. The method for processing Dynamic Vision Sensor (DVS) events according to claim 1 , the acquiring, based on the vision pyramid, the scale-invariant DVS event feature descriptor comprises:
determining a plurality of distance maps for a plurality of events in a layer of the vision pyramid in eight directions;
determining, based on the plurality of distance maps, a plurality of main direction of motions of the plurality of events in the layer; and
extracting the first DVS event feature descriptor based on the plurality of distance maps and the plurality of main direction of motions of the plurality of events in the layer.

6. The method for processing Dynamic Vision Sensor (DVS) events according to claim 5 , wherein the determining the plurality of distance maps comprises:
determining, based on Euclidean distances from the plurality of events in the layer to a closest event in a preset direction and attributes corresponding to the plurality of events, the plurality of distance maps in the eight directions;
wherein the attributes corresponding to the plurality of events comprises:
non-event, negative polarity event and positive polarity event;

wherein the determining the plurality of main direction of motions comprises:
for the plurality of events in the layer, determining a first plurality of directions and the Euclidean distances from DVS events within a neighborhood to a current event and a number of the DVS events within the neighborhood of the current event; and
determining, based on the first plurality of directions, the Euclidean distances and the number of the DVS events, a current main direction of motion of the current event; and

wherein the extracting the first DVS event feature descriptor based on the plurality of distance maps and the plurality of main direction of motions of the plurality of events in the layer comprises:
performing event sampling on the plurality of distance maps of the plurality of events in a current DVS event map in the eight directions in the layer of the vision pyramid, and determining locations of a plurality of sampled events;
by using the plurality of sampled events as centers, rotating and aligning image blocks with a preset pixel size to a main direction of the plurality of main direction of motions of a sampled event in a direction in the layer to obtain the image blocks in the eight directions in the layer;
respectively convoluting the image blocks by a Gaussian kernel with a preset width to obtain a one-dimensional feature vector in the eight directions in the layer; and
extracting the first DVS event feature descriptor based on the one-dimensional feature vector.

7. The method for processing Dynamic Vision Sensor (DVS) events according to claim 2 , wherein the determining, based on the first DVS event feature descriptor, the depth value corresponding to the event comprises:
for the first DVS event map, determining, by using a current event as a center, based on the first DVS event feature descriptor of a plurality of events in image blocks with a preset pixel size, a feature matching cost of the current event in a dual-camera DVS event map at the current moment;
determining a parallax of the current event based on the feature matching cost; and
determining, based on the parallax of the current event, the depth value corresponding to the current event.

8. The method for processing Dynamic Vision Sensor (DVS) events according to claim 2 , wherein the determining, based on the depth value corresponding to the event and the first DVS event feature descriptor, the three-dimensional space pose of the first DVS event map at the current moment comprises:
establishing, based on the first DVS event feature descriptor, a correspondence between a plurality of DVS event maps at different moments;
determining, based on the correspondence between the plurality of DVS event maps at different moments and the depth value respectively corresponding to the a DVS event, a three-dimensional space relative motion matrix between the plurality of DVS event maps at different moments; and
determining, based on the three-dimensional space relative motion matrix between the plurality of DVS event maps at different moments a

Assignees

• Samsung Electronics Co Ltd
• Beijing Samsung Telecom R&D Ct

Inventors

• Shi Feng
• Zou Dong Qing
• Liu wei heng
• Qian De Heng
• Ryu Hyun Surk
• Li Jia
• Xu Jing Tao
• Lee Kyoobin
• Wang Qiang

Classifications

• G06V10/20 Primary

  Image preprocessing · CPC title

• G06T15/20 Primary

  Perspective computation · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title