Digital neuromorphic (NM) sensor array, detector, engine and methodologies

Patent metadata

Field	Value
Publication number	US-10133944-B2
Application number	US-201615386220-A
Country	US
Kind code	B2
Filing date	Dec 21, 2016
Priority date	Dec 21, 2016
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking.

First claim

The invention claimed is:

1. A neuromorphic vision system generating and processing image data, the system comprising:
an image sensor comprising a plurality of photoreceptors each generating image data for generation of spike data, wherein spike data indicates whether an intensity value measured by that photoreceptor exceeds a threshold;
means for generating the spike data based on image data generated by the plurality of photoreceptors, the means for generating the spike data comprising a plurality of computational elements corresponding to the plurality of photoreceptors of the image sensor, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
a digital neuromorphic engine coupled to the means for generating spike data and receiving the generated spike data, the digital neuromorphic engine including one or more processors running software configured to generate, based on the spike data, digital neuromorphic output data that includes velocity vector data,
wherein the velocity vector data represents a velocity space, which is a spatial and temporal representation of the image data generated by the plurality of photoreceptors.

2. The neuromorphic vision system of claim 1 , wherein the velocity vector data are aggregated and associated with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

3. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic vision system compresses high framerate input video data generated by the plurality of photoreceptors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon.

4. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic vision system is incorporated in a stereo neuromorphic pair.

5. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic vision system is incorporated in a compound camera.

6. The neuromorphic vision system of claim 1 , further comprising at least one sensor measuring data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data.

7. The neuromorphic vision system of claim 1 , wherein the plurality of computational elements are implemented in a field-programmable gate array.

8. The neuromorphic vision system of claim 1 , wherein the plurality of computational elements are implemented using complementary metal-oxide-semiconductor technology.

9. The neuromorphic vision system of claim 1 , wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing.

10. A method for generating and processing neuromorphic vision image data, the method comprising:
generating image data using a plurality of photoreceptors, the image data including intensity values measured by the plurality of photoreceptors;
generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold,
wherein generation of the spike data is performed using the plurality of computational elements which correspond to the plurality of photoreceptors, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors,
wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
generating digital neuromorphic output data using a digital neuromorphic engine coupled to the plurality of computational elements, wherein the digital neuromorphic output data includes velocity vector data based on the generated spike data,
wherein the velocity vector data represents a velocity space, which is a spatial and temporal representation of the image data generated by the plurality of photoreceptors.

11. The method for generating and processing neuromorphic vision image data of claim 10 , further comprising aggregating and associating the velocity vector data with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

12. The method for generating and processing neuromorphic vision image data of claim 10 , high framerate input video data generated by the plurality of photoreceptors is compressed by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon.

13. The method for generating and processing neuromorphic vision image data of claim 10 , further comprising measuring data for at least one characteristic using at least one sensor, the at least one characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the plurality of photoreceptors is controlled based on the measured data.

14. The method for generating and processing neuromorphic vision image data of claim 10 , wherein the plurality of computational elements are implemented in a field-programmable gate array.

15. The method for generating and processing neuromorphic vision image data of claim 10 , wherein the plurality of computational elements are implemented using complementary metal-oxide-semiconductor technology.

16. The method for generating and processing neuromorphic vision image data of claim 10 , wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing.

17. A non-transitory computer readable medium including software instructions for performing a method for generating and processing neuromorphic vision image data when the software is run on one or more processors, the method comprising:
generating image data using a plurality of photoreceptors, the image data including intensity values measured by the plurality of photoreceptors;
generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold,
wherein generation of the spike data is performed using the plurality of computational elements which correspond to the plurality of photoreceptors, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors,
wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respectiv

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag

Inventors

• Zink Edmund Dawes
• Hauger Douglas Allen
• Junge Lutz
• Hernandez Gonzalez Luis Marcial
• Gipson Jerramy L
• Vu Anh
• Hempel Martin
• George Nikhil J

Classifications

• G06V10/955

  using specific electronic processors · CPC title

• G06V20/49 Primary

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06V10/20

  Image preprocessing · CPC title

• G06T7/248

  involving reference images or patches · CPC title

• G06N3/049

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title