Biometric sensor fusion to classify vehicle passenger state

Patent metadata

Field	Value
Publication number	US-10867218-B2
Application number	US-201815963697-A
Country	US
Kind code	B2
Filing date	Apr 26, 2018
Priority date	Apr 26, 2018
Publication date	Dec 15, 2020
Grant date	Dec 15, 2020

Abstract

A neural network is used in a vehicle component to determine the stress level or arousal level of a vehicle occupant. Sensors in the vehicle cabin, e.g., the seat, sense biological characteristics of the occupant, e.g., neuro-electrical signals, cardiac characteristics, body temperature and the like. The neural network can compute and classify the emotional state of the occupant in real-time. The vehicle can trigger warnings, indicators and stress counter-measures when the occupant exceeds a threshold. The counter-measures can include visual and audio feedback within the vehicle cabin. The neural network can provide historical occupant emotional states that can be used by the navigation system to avoid travel segments that may trigger undesired emotional states in the occupant.

First claim

What is claimed is:

1. A vehicle system, comprising:
a first occupant sensor to sense central nervous system characteristics of an occupant, wherein the first occupant sensor senses neuroelectric signals;
a second occupant sensor to sense non-central nervous system characteristics of the occupant, wherein the second occupant sensor includes a contactless sensor configured to sense at last one of a sympathetic nervous signal, an autonomic nervous signal, a parasympathetic nervous system signal, or combinations thereof, and wherein the second occupant sensor senses near-infrared spectroscopy signals;
a neural network configured to receive the sensed central nervous system characteristics and the sensed non-central nervous system characteristics and to determine that the occupant has one of a plurality of emotional states based on the sensed central nervous system characteristics and the sensed non-central nervous system characteristics, wherein the plurality of emotional states comprises calm, stressed, and agitated, wherein the neural network includes a first path to process neuroelectric signals, wherein the neural network includes a second path to process near-infrared spectroscopy signals, wherein the first path performs both frequency analysis and temporal analysis of the neuroelectric signals, wherein the first path includes a plurality of first nodes at a cortical and regional signal analysis layer, and wherein the second path includes a plurality of second nodes at a regional activation or deactivation layer; and
a navigation system configured to plan a travel route for a vehicle based on crime data for a road segment of the travel route in response to an occupant having a determined emotional state of stressed or agitated, wherein the navigation system is configured to receive crime data, accident data and occupant stress data for each segment of the planned travel route and when a segment includes a level of any of crime, accident or stress data above an associated threshold, then recalculating the planned travel route to include a different segment with a level of any of crime, accident or stress data below the associated threshold.

2. The vehicle system of claim 1 , wherein the first occupant sensor senses neuroelectric signals, wherein the neural network includes a first path to process neuroelectric signals, wherein the second occupant sensor senses near-infrared spectroscopy signals, and wherein the neural network includes a second path to process near-infrared spectroscopy signals.

3. The vehicle system of claim 2 , wherein the first path performs both frequency analysis and temporal analysis of the neuroelectric signals.

4. The vehicle system of claim 3 , wherein the first path includes a plurality of first nodes at a cortical and regional signal analysis layer; and wherein the second path includes a plurality of second nodes at a regional activation or deactivation layer.

5. The vehicle system of claim 1 , further comprising a seat configured to support the person as an occupant and to be mounted in a vehicle; and wherein the first occupant sensor includes a contactless electro-dermal potential sensor mounted in the seat adjacent a head of the occupant.

6. The vehicle system of claim 5 , wherein the second occupant sensor is a seat-mounted contactless sensor.

7. The vehicle system of claim 1 , wherein the neural network is further configured to (i) determine for a stressed emotional state of the occupant, whether a level of stress of the occupant exceeds a stress threshold, (ii) determine for an agitated emotional state of the occupant, whether a level of agitation of the occupant exceeds an agitation threshold, and (iii) output an indicator signal when the stress threshold or the agitation threshold is exceeded, and wherein the vehicle system further comprises a vehicle-to-occupant interface configured to receive the indicator signal from the neural network and to output an indicator notice within a vehicle cabin to the occupant.

8. The vehicle system of claim 7 , wherein the vehicle-to-occupant interface outputs a neural stimulation signal from an emitter in a seat to reduce the determined emotional state of the occupant to below the threshold.

9. The vehicle system of claim 7 , wherein the indicator notice includes a stored audio signal to calm the occupant below the threshold.

10. The vehicle system of claim 7 , wherein the indicator notice includes a visual image on a display in the vehicle cabin to calm the occupant below the threshold.

11. The vehicle system of claim 7 , wherein the neural network compares the sensed non-central nervous system characteristics of the occupant from the second occupant sensor to stored non-central nervous system characteristics of the occupant to determine if the occupant is in a non-calm state and if the non-calm state is determined, then triggering an occupant intervention action in the vehicle cabin.

12. The vehicle system of claim 1 , wherein the second occupant sensor includes an interior camera mounted in a vehicle cabin directed at a seat to sense the occupant to determine facial expressions.

13. A vehicle system, comprising:
a first occupant sensor to sense central nervous system characteristics of an occupant, wherein the first occupant sensor senses neuroelectric signals;
a second occupant sensor to sense non-central nervous system characteristics of the occupant, wherein the second occupant sensor includes a contactless sensor configured to sense at last one of a sympathetic nervous signal, an autonomic nervous signal, a parasympathetic nervous system signal, or combinations thereof, and wherein the second occupant sensor senses near-infrared spectroscopy signals;
a neural network configured to receive the sensed central nervous system characteristics and the sensed non-central nervous system characteristics, to determine that the occupant has one of a plurality of emotional states based on the sensed central nervous system characteristics and the sensed non-central nervous system characteristics, wherein the plurality of emotional states comprises calm, stressed, and agitated, to output a stress level or an agitation level based on the sensed central nervous system characteristics and the sensed non-central nervous system characteristics, and to store the stress level or agitation level in association with a road segment, wherein the neural network includes a first path to process neuroelectric signals, wherein the neural network includes a second path to process near-infrared spectroscopy signals, wherein the first path performs both frequency analysis and temporal analysis of the neuroelectric signals, wherein the first path includes a plurality of first nodes at a cortical and regional signal analysis layer, and wherein the second path includes a plurality of second nodes at a regional activation or deactivation layer; and
a navigation system configured to plan a travel route for a vehicle based on (i) a stored stress level or a stored agitation level of the occupant for a road segment of the travel route, and (ii) crime data for the road segment of the travel route, wherein the navigation system is configured to receive crime data, accident data and occupant stress data for each segment of the planned travel route and when a segment includes a level of any of crime, accident or stress data above an associated threshold, then recalculating the planned travel route to include a different segment with a level of any of crime, accident or stress data below the associated threshold.

14. The vehicle system of claim 13 , wherein navigation system is configured to receive real-time driving

Assignees

• Lear Corp

Inventors

• Gallagher David
• Migneco Francesco
• Yetukuri Arjun
• Pizana Jasmine
• Stavropoulos Steven T
• Carlsen Pat
• Prakash Abhay

Classifications

• G06V20/593

  Recognising seat occupancy · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• A61B5/24

  Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof · CPC title