System and methodologies for occupant monitoring utilizing digital neuromorphic (NM) data and fovea tracking

Patent metadata

Field	Value
Publication number	US-10387725-B2
Application number	US-201916280462-A
Country	US
Kind code	B2
Filing date	Feb 20, 2019
Priority date	Dec 21, 2016
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking so as to detect and predict movement of a vehicle occupant.

First claim

The invention claimed is:

1. A neuromorphic vision system generating and processing image data, the system comprising:
a digital neuromorphic engine coupled to a spike data generator and receiving spike data generated thereby, wherein the spike data indicates whether a measured intensity value in image data measured by an image sensor exceeds a threshold,
wherein the digital neuromorphic engine includes one or more processors running software configured to generate, based on the spike data, digital neuromorphic output data indicative of the image data gathered by the image sensor and to perform object detection, classification and/or tracking based on the spike data generated by the spike data generator,
wherein the digital neuromorphic engine generates velocity vector and velocity trajectory data based on the generated digital neuromorphic output data and analyzes the velocity vector and velocity trajectory data to identify predictive movement data associated with an occupant action, and
wherein the predictive movement data is compared with continuously monitored velocity trajectory data to predict the occupant action.

2. The neuromorphic vision system of claim 1 , wherein the velocity vector data is used to represent a velocity space, which is a spatial and temporal representation of the image data.

3. The neuromorphic vision system of 1 , wherein the velocity vector data are aggregated and associated with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

4. The neuromorphic vision system of claim 1 , wherein the predictive movement data includes two dimensional silhouettes of objects within a vehicle interior.

5. The neuromorphic vision system of claim 1 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to track head and/or eye tracking of a vehicle occupant.

6. The neuromorphic vision system of claim 1 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to detect at least one of pupil dilation, eye gaze, blink and/or lid detection of a vehicle occupant.

7. The neuromorphic vision system of claim 1 , wherein the velocity vector and velocity trajectory data are analyzed to perform gesture detection for a gesture made by a vehicle occupant.

8. The neuromorphic vision system of claim 1 , wherein vehicle functionality is triggered in response to the comparison of the predictive movement data with the continuously monitored velocity trajectory data.

9. The neuromorphic vision system of claim 8 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates at least one of a driver lowering their chin, rapidly blinking their eyes, closing their eyes half way and the triggered vehicle functionality includes at least one of triggering output of audio, alteration of lighting or tactile output to an vehicle occupant.

10. The neuromorphic vision system of claim 8 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates a driver turning their head away from a windshield of the vehicle and the triggered vehicle functionality includes at least one of outputting audio indicating a presence of oncoming traffic and initiating detection of lane drift by the vehicle.

11. The neuromorphic vision system of claim 8 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates a driver turning their head away from a windshield of the vehicle and the triggered vehicle functionality includes at least one of outputting audio indicating a presence of oncoming traffic and initiating detection of lane drift by the vehicle.

12. A method for monitoring a vehicle interior using neuromorphic vision system to generate and process image data, the method comprising:
receiving, by a digital neuromorphic engine, spike data generated by a spike data generator coupled to the digital neuromorphic engine, wherein the spike data indicates whether a measured intensity value in image data measured by an image sensor exceeds a threshold;
generating, by the digital neuromorphic engine, digital neuromorphic output data, based on the spike data, wherein the digital neuromorphic output data is indicative of the image data gathered by the image sensor;
performing object detection, classification and/or tracking, by the digital neuromorphic engine, based on the spike data generated by the spike data generator;
generating, by the digital neuromorphic engine, velocity vector and velocity trajectory data based on the generated digital neuromorphic output data;
analyzing, by the digital neuromorphic engine, the velocity vector and velocity trajectory data to identify predictive movement data associated with an occupant action; and
comparing, the predictive movement data with continuously monitored velocity trajectory data to predict the occupant action.

13. The method for monitoring a vehicle interior of claim 12 , wherein the velocity vector data is used to represent a velocity space, which is a spatial and temporal representation of the image data.

14. The method for monitoring a vehicle interior of claim 12 , wherein the velocity vector data are aggregated and associated with one another to perform velocity segmentation to identify and differentiate objects within the image data based on their relative motion over frames of image data.

15. The method for monitoring a vehicle interior of claim 12 , wherein the predictive movement data includes two dimensional silhouettes of objects within a vehicle interior.

16. The method for monitoring a vehicle interior of claim 12 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to track head and/or eye tracking of a vehicle occupant.

17. The method for monitoring a vehicle interior of claim 12 , wherein velocity vector and velocity trajectory data are used to perform pattern recognition to detect at least one of pupil dilation, eye gaze, blink and/or lid detection of a vehicle occupant.

18. The method for monitoring a vehicle interior of claim 12 , wherein the velocity vector and velocity trajectory data are analyzed to perform gesture detection for a gesture made by a vehicle occupant.

19. The method for monitoring a vehicle interior of claim 12 , wherein vehicle functionality is triggered in response to the comparison of the predictive movement data with the continuously monitored velocity trajectory data.

20. The method for monitoring a vehicle interior of claim 19 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates at least one of a driver lowering their chin, rapidly blinking their eyes, closing their eyes half way and the triggered vehicle functionality includes at least one of triggering output of audio, alteration of lighting or tactile output to an vehicle occupant.

21. The method for monitoring a vehicle interior of claim 19 , wherein the comparison of the predictive movement data is compared with continuously monitored velocity trajectory data indicates a driver turning their head away from a windshield of the vehicle and the triggered vehicle functionality includes at least one of outputting audio indicating a presence of oncoming traffic and initiating detection of lane

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag

Inventors

• Zink Edmund Dawes
• Hauger Douglas Allen
• Junge Lutz
• Hernandez Gonzalez Luis Marcial
• Gipson Jerramy L
• Vu Anh
• Hempel Martin
• George Nikhil J

Classifications

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N25/76

  Addressed sensors, e.g. MOS or CMOS sensors · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title