Digital neuromorphic (NM) sensor array, detector, engine and methodologies

Patent metadata

Field	Value
Publication number	US-10387741-B2
Application number	US-201816194462-A
Country	US
Kind code	B2
Filing date	Nov 19, 2018
Priority date	Dec 21, 2016
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking.

First claim

The invention claimed is:

1. A neuromorphic vision system generating and processing image data, the system comprising:
a composite image sensor comprised of a plurality of photoreceptors for generating image data indicating whether an intensity value measured by each photoreceptor exceeds a threshold;
a plurality of computational elements corresponding to the plurality of photoreceptors, wherein each of the plurality of computational elements generates image spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
a digital neuromorphic engine including one or more processors running software configured to generate, based on the generated image spike data, digital neuromorphic output data that includes velocity vector data representing a velocity space that is a spatial and temporal representation of the image data.

2. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine aggregates and associates the velocity vector data to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data.

3. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine compresses high framerate input video data generated by the plurality of photoreceptors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon.

4. The neuromorphic vision system of claim 1 , wherein the neuromorphic vision system is incorporated in a stereo neuromorphic pair or a compound camera.

5. The neuromorphic vision system of claim 1 , further comprising at least one sensor measuring data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor.

6. The neuromorphic vision system of claim 1 , wherein the plurality of computational elements are implemented in a field-programmable gate array or using complementary metal-oxide-semiconductor technology.

7. The neuromorphic vision system of claim 1 , wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing by the digital neuromorphic engine.

8. A method for generating and processing neuromorphic vision image data, the method comprising:
generating image data using a plurality of photoreceptors;
generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
generating digital neuromorphic output data using a digital neuromorphic engine coupled to the plurality of computational elements, wherein the digital neuromorphic output data includes velocity vector data generated based on the generated spike data and representing a velocity space that is a spatial and temporal representation of the image data generated by the plurality of photoreceptors.

9. The method of claim 8 , further comprising aggregating and associating the velocity vector data by the digital neuromorphic engine to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data.

10. The method of claim 8 , further comprising compressing, by the digital neuromorphic engine, high framerate input video data generated by the plurality of photoreceptors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon.

11. The method of claim 8 , further comprising measuring, by at least one sensor, data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor.

12. The method of claim 8 , wherein the plurality of computational elements are implemented in a field-programmable gate array or using complementary metal-oxide-semiconductor technology.

13. The method of claim 8 , wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing by the digital neuromorphic engine.

14. A non-transitory computer readable medium including software instructions for performing a method for generating and processing neuromorphic vision image data when the software is run on one or more processors, the method comprising:
generating image data using a plurality of photoreceptors;
generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
generating digital neuromorphic output data using a digital neuromorphic engine coupled to the plurality of computational elements, wherein the digital neuromorphic output data includes velocity vector data generated based on the generated spike data and representing a velocity space that is a spatial and temporal representation of the image data generated by the plurality of photoreceptors.

15. The non-transitory computer readable medium of claim 14 , further comprising aggregating and associating the velocity vector data by the digital neuromorphic engine to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data.

16. The non-transitory computer readable medium of claim 14 , further comprising compressing, by the digital neuromorphic engine, high framerate input video data generated by the plurality of photoreceptors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon.

17. The non-transitory computer readable medium of claim 14 , further comprising measuring, by at least one sensor, data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor.

18. The non-transitory computer readable m

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag

Inventors

• Zink Edmund Dawes
• Hauger Douglas Allen
• Junge Lutz
• Hernandez Gonzalez Luis Marcial
• Gipson Jerramy L
• Vu Anh
• Hempel Martin
• George Nikhil J

Classifications

• G06V10/955

  using specific electronic processors · CPC title

• G06V20/49 Primary

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06V10/20

  Image preprocessing · CPC title

• G06T7/215

  Motion-based segmentation · CPC title

• G06N3/063

  using electronic means · CPC title