In-vehicle multimodal violence detection for shared vehicles and self-driving shuttles

Patent metadata

Field	Value
Publication number	US-12115995-B2
Application number	US-202217829643-A
Country	US
Kind code	B2
Filing date	Jun 1, 2022
Priority date	Jun 1, 2022
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

An in-vehicle violence detection system includes a speech and non-speech audio recognition module capturing occupant threat words and non-speech audio events. In-vehicle accelerometers generate data analyzed in a shaking movement recognition module. A heart rate and breathing rate detection module measures physiological changes in occupant heart rates and breathing rhythms. An in-vehicle semantic scene recognition module analyzes occupant non-verbal interactions. Occupant threat indicators including an audible threat indicator are generated by the speech and non-speech audio event recognition module. Visual threat indicators are generated by the in-vehicle semantic scene recognition module. Physiological threat indicators are generated by the heart rate and breathing rate detection module. Vibration-based threat indicators are generated by the shaking movement recognition module. The four threat indicators are fused to estimate a threat level, consolidated by incorporating contextual information including a vehicle location and time. Different actions are taken based on the consolidated threat level.

First claim

What is claimed is:

1. An in-vehicle multimodal violence detection system, comprising:
a speech and non-speech audio event recognition module capturing threat words and non-speech audio events of occupants of an automobile vehicle;
multiple in-vehicle accelerometers generating in-vehicle accelerometer data analyzed in a shaking movement recognition module;
one or more in-vehicle radar devices detecting heart rates and breathing rates of vehicle occupants, the system directing signals from the in-vehicle radar devices to a heart rate and breathing rate detection module, the heart rate and breathing detection module measuring physiological changes in heart rates and breathing rhythms of the occupants and analyzing sudden changes in heart rates and breathing rhythms;
one or more in-vehicle cameras capturing occupant positions and movements over time, the system directing signals from the one or more in-vehicle cameras to an in-vehicle semantic scene recognition module that captures and analyzes non-verbal interactions between the occupants;
one or more occupant threat indicators including an audible threat indicator generated by the speech and non-speech audio event recognition module, a visual threat indicator generated by the in-vehicle semantic scene recognition module, a physiological threat indicator generated by the heart rate and breathing rate detection module and a vibration-based threat indicator generated by the shaking movement recognition module;
a threat level determination unit initiated by a time interval time-out signal if a predetermined time interval is exceeded prior to receiving a response from any one of the occupants;
an active threat signal generated and forwarded to the threat level determination unit if any one of the occupants confirm that an active threat is present;
the threat level determination unit identifying if the active threat should be categorized as one of a “low threat”, a “medium threat” or a “high threat”, wherein:
identification of the “low threat” activates a vehicle horn and a vehicle warning flasher;
identification of the “medium threat” activates the vehicle horn and the vehicle warning flasher and activates a vehicle brake assist system and notification of the medium threat to an outside source; and
identification of the “high threat” activates the vehicle horn and the vehicle warning flasher, activates the vehicle brake assist system and forwards a request to the outside source to initiate an emergency assistance request.

2. The in-vehicle multimodal violence detection system of claim 1 , including a priori knowledge defining a history record of violence applied to determine a threat level.

3. The in-vehicle multimodal violence detection system of claim 2 , including an outside source providing the a priori information.

4. The in-vehicle multimodal violence detection system of claim 2 , wherein the a priori information includes an area of travel wherein known violence has occurred and a past history record of different “normal” versus “threat” events.

5. The in-vehicle multimodal violence detection system of claim 1 , including a picture compilation module identifying vectors combining an output signal from the speech and non-speech audio event recognition module, the shaking movement recognition module, the heart rate and breathing rate detection module and the in-vehicle semantic scene recognition module and assigning predetermined thresholds distinguishing between a normal event and a threat event.

6. The in-vehicle multimodal violence detection system of claim 5 , including a threat evaluation module analyzing the threat event using audible, visual, physiological and vibration-based indicators, and an a priori knowledge including a history record of violence and contextual information having a location and a time of day to determine a threat level.

7. The in-vehicle multimodal violence detection system of claim 6 , including a threat assessment unit receiving the threat level and confirming if an active threat is present and generating an active threat signal.

8. The in-vehicle multimodal violence detection system of claim 7 , wherein after generating the active threat signal a threat confirmation request is generated and forwarded to a threat timer unit.

9. The in-vehicle multimodal violence detection system of claim 8 , including:
a confirmation request visually and audibly presented to the occupants and the predetermined time interval allowed for at least one of the occupants to confirm if the active threat is present in the automobile vehicle is set.

10. A method to perform in-vehicle multimodal violence detection, comprising:
capturing threat words and non-speech audio events of occupants of an automobile vehicle using a speech and non-speech audio event recognition module;
analyzing in-vehicle accelerometer data in a shaking movement recognition module;
identifying, with one or more in-vehicle radar devices, sudden physiological changes in heart rates and breathing rhythms of the occupants in a heart rate and breathing rate detection module, the one or more in-vehicle radar devices detecting heart rates and breathing rates of vehicle occupants;
directing signals from the in-vehicle radar devices to the heart rate and breathing rate detection module;
capturing, with one or more in-vehicle cameras, occupant positions and movements over time;
directing signals from the one or more in-vehicle cameras to an in-vehicle semantic scene recognition module;
analyzing, with in-vehicle semantic scene recognition module, non-verbal interactions between the occupants; and
generating one or more occupant threat indicators including audible, visual, physiological and vibration-based indicators, including:
initiating, by a time interval time-out signal, a threat level determination unit if a predetermined time interval is exceeded prior to receiving a response from any one of the occupants;
generating and forwarding an active threat signal to the threat level determination unit if any one of the occupants confirm that an active threat is present;
identifying, by the threat level determination unit, if the active threat should be categorized as one of a “low threat”, a “medium threat” or a “high threat”, and:
activating a vehicle horn and a vehicle warning flasher upon identification of the “low threat”;
activating the vehicle horn and the vehicle warning flasher and activating a vehicle brake assist system and notifying an outside source of the “medium threat” upon identification of the “medium threat”; and
activating the vehicle horn and the vehicle warning flasher, activating the vehicle brake assist system and forwarding a request to the outside source to initiate an emergency assistance request upon identification of the “high threat”.

11. The method of claim 10 , further including applying a priori knowledge including a history record of violence to determine a threat level.

12. The method of claim 11 , further including applying contextual information including a vehicle location and a time of day from the a priori knowledge.

13. The method of claim 10 , further including incorporating ride information and occupant information from a booking database and analyzing the ride information and the occupant information to distinguish differences between the occupants.

14. The method of claim 10 , further including identifying shaking movements of the occupants of the automobile vehicle using the in-vehicle accelerometer data.

15. The method of claim 10 , further including applying inarticulate sounds and sounds accompanying threat or abnormal behaviors i

Assignees

• Gm Global Tech Operations Llc

Inventors

• Khamis Alaa M
• Tong Wei
• Aluru Sai Vishnu
• Mudalige Upali P
• Daruri Sravan

Classifications

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• A61B5/165

  Evaluating the state of mind, e.g. depression, anxiety · CPC title

• A61B5/0205

  Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition · CPC title

• B60W2040/089

  Driver voice · CPC title

• B60W30/18109

  Braking · CPC title