System and methodologies for occupant monitoring utilizing digital neuromorphic (NM) data and fovea tracking

Patent metadata

Field	Value
Publication number	US-10235565-B2
Application number	US-201715726883-A
Country	US
Kind code	B2
Filing date	Oct 6, 2017
Priority date	Dec 21, 2016
Publication date	Mar 19, 2019
Grant date	Mar 19, 2019

Abstract

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking so as to detect and predict movement of a vehicle occupant.

First claim

The invention claimed is:

1. A neuromorphic vision system generating and processing image data, the system comprising:
an image sensor comprising a plurality of photoreceptors each generating image data for generation of spike data, wherein spike data indicates whether an intensity value measured by that photoreceptor exceeds a threshold;
a spike data generator that generates the spike data based on the image data generated by the plurality of photoreceptors, the spike data generator comprising a plurality of computational elements corresponding to the plurality of photoreceptors of the image sensor, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
a digital neuromorphic engine coupled to the spike data generator and receiving the generated spike data, the digital neuromorphic engine including one or more processors running software configured to generate, based on the spike data, digital neuromorphic output data indicative of the image data gathered by the image sensor and to perform object detection, classification and/or tracking based on the spike data generated by the spike data generator,
wherein the digital neuromorphic engine generates velocity vector and velocity trajectory data based on the generated digital neuromorphic output data and analyzes the velocity vector and velocity trajectory data to identify predictive movement data associated with an occupant action, and
wherein the predictive movement data is compared with continuously monitored velocity trajectory data to predict the occupant action.

2. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine generates composite spike data using neighborhood image data including the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor.

3. The neuromorphic vison system of claim 1 , wherein the velocity vector data is used to represent a velocity space, which is a spatial and temporal representation of the image data generated by the plurality of photodetectors.

4. The neuromorphic vision system of 3 , wherein the velocity vector data are aggregated and associated with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

5. The neuromorphic vision system of claim 1 , wherein the digital neuromorphic engine generates compound foveas and performs velocity segmentation using the compound foveas to generate velocity vector and velocity trajectory data.

6. The neuromorphic vision system of claim 1 , wherein the predictive movement data includes two dimensional silhouettes of objects within a vehicle interior.

7. The neuromorphic vision system of claim 1 , wherein the comparison with the predictive movement data includes rotation of a three-dimensional model to match two-dimensional, continuously monitored velocity trajectory data.

8. The neuromorphic vision system of claim 1 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to track head and/or eye tracking of a vehicle occupant.

9. The neuromorphic vision system of claim 1 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to detect at least one of pupil dilation, eye gaze, blink and/or lid detection of a vehicle occupant.

10. The neuromorphic vision system of claim 1 , wherein the velocity vector and velocity trajectory data are analyzed to perform gesture detection for a gesture made by a vehicle occupant.

11. The neuromorphic vision system of claim 1 , wherein vehicle functionality is triggered in response to the comparison of the predictive movement data with the continuously monitored velocity trajectory data.

12. The neuromorphic vision system of claim 10 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates at least one of a driver lowering their chin, rapidly blinking their eyes, closing their eyes half way and the triggered vehicle functionality includes at least one of triggering output of audio, alteration of lighting or tactile output to an vehicle occupant.

13. The neuromorphic vision system of claim 10 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates a driver turning their head away from a windshield of the vehicle and the triggered vehicle functionality includes at least one of outputting audio indicating a presence of oncoming traffic and initiating detection of lane drift by the vehicle.

14. The neuromorphic vision system of claim 10 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates a driver turning their head away from a windshield of the vehicle and the triggered vehicle functionality includes at least one of outputting audio indicating a presence of oncoming traffic and initiating detection of lane drift by the vehicle.

15. A method for monitoring a vehicle interior using neuromorphic vision system to generate and process image data, the method comprising:
generating image data using an image sensor comprising a plurality of photoreceptors;
generating spike data using a spike data generator based on the image data generated by the plurality of photoreceptors, wherein the spike data indicates whether an intensity value measured by each photoreceptor exceeds a threshold, wherein the spike data generator comprises a plurality of computational elements corresponding to the plurality of photoreceptors of the image sensor, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and
generating digital neuromorphic output data using a digital neuromorphic engine based on the spike data, wherein the digital neuromorphic output data is indicative of the image data gathered by the image sensor;
performing object detection, classification and/or tracking based on the spike data generated by the spike data generator;
generating velocity vector and velocity trajectory data based on the generated digital neuromorphic output data; and
analyzing the velocity vector and velocity trajectory data to identify predictive movement data associated with an occupant action, wherein the predictive movement data is compared with continuously monitored velocity trajectory data to predict the occupant action.

16. The monitoring method of claim 15 , further comprising generating composite spike data using neighborhood image data including the image data generated by at least two of the plurality of photoreceptors, wherein the at least two of the plurality of photoreceptors includes the respective corresponding photoreceptor and a photoreceptor neighboring the respec

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag
• Posrche AG

Inventors

• Zink Edmund Dawes
• Hauger Douglas Allen
• Junge Lutz
• Hernandez Gonzalez Luis Marcial
• Gipson Jerramy L
• Vu Anh
• Hempel Martin
• George Nikhil J

Classifications

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N25/76

  Addressed sensors, e.g. MOS or CMOS sensors · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title