System and method for real-time feedback of remote collaborative communication

Patent metadata

Field	Value
Publication number	US-12367876-B2
Application number	US-202218065248-A
Country	US
Kind code	B2
Filing date	Dec 13, 2022
Priority date	Dec 13, 2022
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

A system and method for providing real-time feedback of remote collaborative communication between users includes extracting speech-related features and physiological features from at least one of the users and using these features to determine a stress state of at least one user. In response to the determined stress state audio signals may be processed to manipulate one or more vocal features of the speech supplied from another user, and/or at least one device may supply feedback to another user that provides suggestions as to how to manipulate or more of their vocal features.

First claim

What is claimed is:

1. A system for providing real-time feedback of remote collaborative communication between a first user and a second user, the system comprising:
a non-transitory computer-readable medium storing instructions; and
a processing system, including at least one specialized processor, that is configured to execute the stored instructions to:
extract a plurality of first speech-related features from first audio signals generated in response to speech supplied from the first user;
extract a plurality of first physiological features from the first audio signals, first physiological data generated in response to physiological activity of the first user, or both, wherein the first physiological data is obtained from one or more first physiological sensors;
extract a plurality of second speech-related features from second audio signals generated response to speech supplied from the second user;
extract a plurality of second physiological features from the second audio signals, second physiological data generated in response to physiological activity of the second user, or both, wherein the second physiological data is obtained from one or more second physiological sensors;
process the first and second speech-related features to determine if the speech from the first and second users exhibits positive or negative vocal entrainment;
process the first and second physiological features to determine if physiological states of the first and second users are aligned or misaligned; and
generate commands that cause at least one device to supply real-time, adaptive multimodal feedback that indicates:
(i) whether the speech from the first and second users exhibits positive or negative vocal entrainment, and
(ii) whether the physiological activity of the first and second users is aligned or misaligned.

2. The system of claim 1 , wherein the processing system is further configured to:
process the first and second physiological features to determine if the physiological states of the first and second users are increasing or decreasing; and
generate commands that cause the at least one device to supply feedback that indicates if the physiological states of the first and second users are increasing or decreasing.

3. The system of claim 1 , further comprising:
a first microphone configured to receive the speech supplied from the first user and, in response thereto, supply the first audio signals to the processing system; and
a second microphone configured to receive the speech supplied from the second user and, in response thereto, supply the second audio signals to the processing system.

4. The system of claim 1 , further comprising:
the plurality of first physiological sensors disposed on the first user and configured to supply the first physiological data to the processing system; and
the plurality of second physiological sensors disposed on the second user and configured to supply the second physiological data to the processing system.

5. The system of claim 1 , further comprising:
a first human-machine interface coupled to receive the commands from the processing system; and
a second user human-machine interface coupled to receive the commands from the processing system.

6. The system of claim 5 , further comprising:
a third human-machine interface disposed remote from the first and second human-machine interfaces, the third human-machine interface coupled to receive the commands from the processing system.

7. The system of claim 1 , wherein the first and second speech-related features that the processing system is configured to extract comprise: pitch, speech rate, intensity, jitter, and shimmer.

8. The system of claim 1 , wherein the first and second physiological features that the processing system is configured to extract comprise one or more of: heart rate, breath rate, oxygen saturation level, pupil diameter, and galvanic skin response.

9. The system of claim 1 , wherein the processing system is configured to correlate the physiological state of the first and second users to a stress state of the first and second user, respectively, and determine if the stress state of the first and second users is aligned or misaligned.

10. The system of claim 1 , wherein the processing system comprises:
a first local processor coupled to receive and process the first audio signals and the first physiological data and to transmit the first speech-related features and the first physiological features;
a second local processor coupled to receive and process the second audio signals and the second physiological data and to transmit the second speech-related features and the second physiological features; and
a cloud processor coupled to receive and process (i) the first and second speech-related features to determine if the speech from the first and second users exhibits positive or negative vocal entrainment and (ii) the first and second physiological features to determine if the physiological states of the first and second users are aligned or misaligned.

Assignees

• Honeywell Int Inc

Inventors

• Lubold Nichola
• Finseth Tor

Classifications

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title

• G10L17/26

  Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices · CPC title

• A61B5/4836

  Diagnosis combined with treatment in closed-loop systems or methods (A61B5/0036 takes precedence) · CPC title

• A61B5/7267

  involving training the classification device · CPC title

• G10L25/30

  using neural networks · CPC title