Vector engine and methodologies using digital neuromorphic (NM) data

Patent metadata

Field	Value
Publication number	US-10229341-B2
Application number	US-201715619906-A
Country	US
Kind code	B2
Filing date	Jun 12, 2017
Priority date	Dec 21, 2016
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking.

First claim

The invention claimed is:

1. A neuromorphic vision system generating and processing image data, the system comprising:
an image sensor comprising a plurality of photoreceptors each generating image data for generation of spike data, wherein spike data indicates whether an intensity value measured by that photoreceptor exceeds a threshold;
a spike data generator that generates the spike data based on the image data generated by the plurality of photoreceptors, the spike data generator comprising a plurality of computational elements corresponding to the plurality of photoreceptors of the image sensor, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
a digital neuromorphic engine coupled to the spike data generator and receiving the generated spike data, the digital neuromorphic engine including one or more processors running software configured to generate, based on the spike data, digital neuromorphic output data that includes velocity vector data indicative of the image data gathered by the image sensor, wherein the velocity vector data represents a velocity space, which is a spatial and temporal representation of the image data generated by the plurality of photoreceptors and to perform object detection, classification and/or tracking based on the velocity vector data.

2. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine performs velocity vector and velocity trajectory generation to generate the velocity vector data.

3. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine performs image segmentation based on high density velocity vectors.

4. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine generates and analyzes spatial temporal patterns N×5D (x,y,Vx,Vy,Vz) and N×5D (x,y,Vx,Vy,t).

5. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine generates composite spike data using neighborhood image data including the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor.

6. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine generates compound foveas and performs velocity segmentation using the compound foveas.

7. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine computes velocity trajectories for fovea pixels.

8. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine computes velocity vectors by rotating velocity space and using previous frames of image data to increase the temporal and spatial resolution of spikes within the spike data.

9. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine uses spread functions to optimize search for velocity vectors.

10. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine performs a double sparsity approach to compute velocity vectors, and computing velocity trajectories using affine transformations.

11. The neuromorphic vision system of claim 1 , wherein the velocity vector data are aggregated and associated with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

12. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic vision system is incorporated in a compound camera.

13. A neuromorphic imaging method that generates and processes neuromorphic image data, the method comprising:
generating image data using an image sensor comprising a plurality of photoreceptors that each generate image data for generation of spike data, wherein spike data indicates whether an intensity value measured by that photoreceptor exceeds a threshold;
generating the spike data based on the image data generated by the plurality of photoreceptors, wherein the generation of the spike data is performed using a plurality of computational elements corresponding to the plurality of photoreceptors of the image sensor, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor;
generating, based on the spike data, digital neuromorphic output data that includes velocity vector data indicative of the image data gathered by the image sensor, wherein the velocity vector data represents a velocity space, which is a spatial and temporal representation of the image data generated by the plurality of photoreceptors; and
performing object detection, classification and/or tracking based on the velocity vector data.

14. The neuromorphic imaging method of claim 13 , wherein the digital neuromorphic output data includes the velocity vector data and velocity trajectory data.

15. The neuromorphic imaging method of claim 13 , wherein the digital neuromorphic output data includes image segmentation data generated based on high density velocity vectors.

16. The neuromorphic imaging method of claim 13 , wherein the digital neuromorphic output data is generated by analyzing spatial temporal patterns N×5D (x,y,Vx,Vy,Vz) and N×5D (x,y,Vx,Vy,t).

17. The neuromorphic imaging method of claim 13 , further comprising generating composite spike data using neighborhood image data including the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor.

18. The neuromorphic imaging method of claim 13 , further comprising generating compound foveas and performing velocity segmentation on the neuromorphic output data using the compound foveas.

19. The neuromorphic imaging method of claim 13 , further comprising computing velocity trajectories for fovea pixels included in the neuromorphic output data.

20. The neuromorphic imaging method of claim 13 , wherein the velocity vector data includes velocity vectors computed by rotating velocity space and using previous frames of image data to increase the temporal and spatial resolution of generated spikes within the spike data.

21. The neuromorphic imaging method of claim 13 , wherein the digital neuromorphic output data is generated, in part, using spread functions to optimize search for velocity vectors within the digital neuromorphic data.

22. The neuromorphic imaging method of claim 13 , wherein the digital neuromorphic output data is generated, in part, by performing a double sparsity approach to compute velocity vectors, and computing velocity trajectories using affine transformations.

23. The neuromorphic vision method of 13 , wherein the velocity vector

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag

Inventors

• Zink Edmund Dawes
• Hauger Douglas Allen
• Junge Lutz
• Hernandez Gonzalez Luis Marcial
• Gipson Jerramy L
• Vu Anh
• Hempel Martin
• George Nikhil J

Classifications

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V10/20

  Image preprocessing · CPC title

• G06T2207/10024

  Color image · CPC title