Systems and methods for detecting symptoms of occupant illness

Patent metadata

Field	Value
Publication number	US-12230039-B2
Application number	US-202318385598-A
Country	US
Kind code	B2
Filing date	Oct 31, 2023
Priority date	Oct 23, 2020
Publication date	Feb 18, 2025
Grant date	Feb 18, 2025

Abstract

Systems and methods for detecting symptoms of occupant illness is disclosed herein. In embodiments, a storage is configured to maintain a visualization application and data from one or more sources, such as an image source. A processor is in communication with the storage and a user interface. The processor is programmed to receive data from the one or more sources, execute human-detection models based on the received data, execute activity-recognition models to recognize symptoms of illness based on the data from the one or more sources, determine a location of the recognized symptoms, and execute a visualization application to display information in the user interface. The visualization application can show a background image with an overlaid image that includes an indicator for each location of recognized symptom of illness. Additionally, data from the audio source, image source, and/or radar source can be fused.

First claim

What is claimed is:

1. A system for detecting symptoms of occupant illness, the system comprising:
user interface;
a storage configured to maintain a visualization application and image data from an image source; and
a processor in communication with the storage and the user interface, and programmed to:
receive the image data from the image source, the image data including a background image associated with an area that occupants are occupying,
execute a human-detection model configured to detect the occupants within the image data,
execute an activity-recognition model configured to recognize image-based symptoms of illness in the detected occupants within the image data based on movements of the detected occupants,
determine a location of the recognized symptoms of illness utilizing the image data from the image source, and
execute the visualization application to display, in the user interface, an overlaid image overlaid onto the background image, the overlaid image including, for each location of recognized symptoms of illness, an indicator displaying information that the recognized symptoms of illness occurred at that location, wherein the overlaid image includes a color-coded heat map that varies in intensity corresponding to a number of recognized symptoms of illness at that location.

2. The system of claim 1 , wherein the processor is further programmed to utilize a convolutional network to extract relevant features from the image data, and send the extracted relevant features to the activity-recognition model for recognizing the symptoms of illness.

3. The system of claim 1 , wherein the processor is further programmed to aggregate the recognized symptoms of illness over time to determine a time-series aggregation, wherein the indicator at each location changes based on the time-series aggregation at that location.

4. The system of claim 1 , wherein the processor is further programmed to:
receive audio data from an audio source,
execute one or more models to determine audible symptoms of illness based on the audio data,
fuse the audible symptoms of illness with the image-based symptoms of illness, and
execute the visualization application based on the fused audible symptoms of illness and the image-based symptoms of illness.

5. The system of claim 4 , wherein the fusion of the audible symptoms of illness with the image-based symptoms of illness occurs prior to the execution of the activity-recognition model such that the activity-recognition model is configured to utilize both audible symptoms of illness and image-based symptoms of illness to recognize symptoms of illness in the detected occupants.

6. The system of claim 4 , wherein the fusion of the audible symptoms of illness with the image-based symptoms of illness occurs subsequent to the execution of the activity-recognition model and prior to the execution of the visualization application.

7. The system of claim 1 , wherein the processor is further programmed to:
receive radar data from a radar source,
execute a human-detection model configured to detect the occupants based on the radar data,
execute an activity-recognition model or vital-signs-recognition model configured to recognize radar-based symptoms of illness in the detected occupants based on the radar data,
fuse the recognized radar-based symptoms of illness with the image-based symptoms of illness, and
execute the visualization application based on the fused radar-based symptoms of illness and image-based symptoms of illness.

8. The system of claim 7 , wherein the processor is further programmed to:
receive audio data from an audio source,
execute one or more models to determine audible symptoms of illness based on the audio data,
fuse the audible symptoms of illness with the image-based symptoms of illness and the radar-based symptoms of illness, and
execute the visualization application based on the fused audible symptoms of illness, image-based symptoms of illness, and radar-based symptoms of illness.

9. The system of claim 1 , wherein the processor is further programmed to:
process the image data to detect a presence of a body of one of the occupants.

10. The system of claim 1 , wherein the activity-recognition model is further configured to process spatio-temporal features in the image data corresponding to the detected occupants in order to recognize the image-based symptoms of illness.

11. A method for detecting symptoms of occupant illness, the method comprising:
receiving image data from an image source, the image data including a background image associated with an area that occupants are occupying;
executing a human-detection model configured to detect the occupants within the image data;
executing an activity-recognition model configured to recognize image-based symptoms of illness in the detected occupants within the image data based on movements of the detected occupants;
determine a location of the recognized symptoms of illness utilizing the image data from the image source; and
execute a visualization application to display, in a user interface, an overlaid image overlaid onto the background image, the overlaid image including, for each location of recognized symptoms of illness, an indicator displaying information that the recognized symptoms of illness occurred at that location, wherein the overlaid image includes a color-coded heat map that varies in intensity corresponding to a number of recognized symptoms of illness at that location.

12. The method of claim 11 , further comprising:
utilizing a convolutional network to extract relevant features from the image data, and sending the extracted relevant features to the activity-recognition model for recognizing the symptoms of illness.

13. The method of claim 11 , further comprising:
aggregating the recognized symptoms of illness over time to determine a time-series aggregation, wherein the indicator at each location changes based on the time-series aggregation at that location.

14. The method of claim 11 , further comprising:
receiving audio data from an audio source;
executing one or more models to determine audible symptoms of illness based on the audio data;
fusing the audible symptoms of illness with the image-based symptoms of illness; and
executing the visualization application based on the fused audible symptoms of illness and the image-based symptoms of illness.

15. The method of claim 14 , wherein the fusion of the audible symptoms of illness with the image-based symptoms of illness occurs prior to the execution of the activity-recognition model such that the activity-recognition model is configured to utilize both audible symptoms of illness and image-based symptoms of illness to recognize symptoms of illness in the detected occupants.

16. The method of claim 14 , wherein the fusion of the audible symptoms of illness with the image-based symptoms of illness occurs subsequent to the execution of the activity-recognition model and prior to the execution of the visualization application.

17. The method of claim 11 , further comprising:
receiving radar data from a radar source;
executing a human-detection model configured to detect the occupants based on the radar data;
executing an activity-recognition model or vital-signs-recognition model configured to recognize radar-based symptoms of illness in the detected occupants based on the radar data;
fusing the recognized radar-based symptoms of illness with the image-based symptoms of illness; and
executing the visualization application based on the fused radar-based symptom

Assignees

• Bosch Gmbh Robert

Inventors

• Munir Sirajum
• Das Samarjit
• Zeng Yunze
• Jain Vivek

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title