Multimedia conferencing system for determining participant engagement

Patent metadata

Field	Value
Publication number	US-10586131-B2
Application number	US-201715646823-A
Country	US
Kind code	B2
Filing date	Jul 11, 2017
Priority date	Jul 11, 2017
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

A method for operating a teleconference includes receiving a first video and audio signal from a first teleconference participant and receiving a second video and audio signal from a second teleconference participant. The method further includes relaying the first video and audio signal to the second teleconference participant and relaying the second video and audio signal to the first teleconference participant. The method additionally includes analyzing the first video and audio signal using a plurality of behavior analytics to determine a level of participant engagement of the first teleconference participant. The method further includes determining that the first teleconference participant is not engaged when the determined level of participant engagement is below a predetermined threshold.

First claim

What is claimed is:

1. A method for operating a teleconference, comprising:
receiving a first video and audio signal from a first teleconference participant;
receiving a second video and audio signal from a second teleconference participant;
relaying the first video and audio signal to the second teleconference participant and relaying the second video and audio signal to the first teleconference participant;
analyzing the first video and audio signal using a plurality of behavior analytics to determine a level of participant engagement of the first teleconference participant, wherein the plurality of behavior analytics includes determining an emotional state of the first teleconference participant; and
determining that the first teleconference participant is not engaged when the determined level of participant engagement is below a predetermined threshold.

2. A method for operating a teleconference, comprising:
receiving a first video and audio signal from a first teleconference participant;
receiving a second video and audio signal from a second teleconference participant;
relaying the first video and audio signal to the second teleconference participant and relaying the first video and audio signal to the first teleconference participant;
analyzing the first video and audio signal using a plurality of behavior analytics to determine a level of participant engagement of the first teleconference participant; and
determining that the first teleconference participant is not engaged when the determined level of participant engagement is below a predetermined threshold, wherein when it is determined that the first teleconference participant is not engaged, relaying of the first video and audio signal to the second teleconference participant is temporarily discontinued.

3. The method of claim 2 , further comprising:
continuing to analyze the first video and audio signal using the plurality of behavior analytics to determine the level of participant engagement of the first teleconference participant;
determining that the first teleconference participant has reengaged when the determined level of participant engagement is above the predetermined threshold; and
resuming the relaying of the first video and audio signal to the second teleconference participant when it is determined that the first teleconference participant has reenaged.

4. The method of claim 1 , wherein at least one additional video and audio signal is received from at least one additional teleconference participant, the first and second video and audio signals are relayed to the at least one additional teleconference participant, and the at least one additional video and audio signal is relayed to each of the first and second teleconference participants.

5. The method of claim 1 , further comprising:
analyzing the second video and audio signal using the plurality of behavior analytics to determine a level of participant engagement of the second teleconference participant; and
determining that the second teleconference participant is not engaged when the determined level of participant engagement of the second teleconference participant is below the predetermined threshold.

6. The method of claim 5 , wherein when it is determined that the second teleconference participant is not engaged, relaying of the second video and audio signal to the first teleconference participant is temporarily discontinued.

7. The method of claim 1 , wherein when it is determined that the first teleconference participant is not engaged, relaying of the first video and audio signal to the second teleconference participant is performed at a reduced bandwidth.

8. The method of claim 1 , wherein the plurality of behavior analytics further includes:
determining whether the first teleconference participant is talking to someone or something other than the teleconference;
determining whether the first teleconference participant has turned away from the teleconference;
determining whether the first teleconference participant is engaged in operating, or is receiving a ring from, a telephone unrelated to the teleconference; or
determining a level of alertness of the first teleconference participant.

9. The method of claim 1 , wherein the plurality of behavior analytics used to determine the level of participant engagement of the first teleconference participant are generated using one or more machine learning techniques.

10. The method of claim 9 , wherein one or more of the plurality of behavior analytics is refined using input from the first or second teleconference participants during the teleconference.

11. A method for training a classifier for determining whether a teleconference participant is engaged in a teleconference, comprising:
receiving a plurality of video and audio signals from one or more teleconference participants participating in one or more teleconferences;
receiving classification data classifying the one or more teleconference participants as engaged or not engaged, based on their observed behavior; and
using one or more computer learning techniques to train the classifier for determining whether a teleconference participant is engaged in a teleconference, using the receiving a plurality of video and audio signals from one or more teleconference participants participating in one or more teleconferences and the received classification data, as training data, wherein the trained classifier is used to determine whether two or more participants in a subsequent teleconference are engaged.

12. The method of claim 11 , wherein the trained classifier is refined during the subsequent teleconference based on input from the two or more participants.

13. A computer program product for operating a teleconference, the computer program product comprising non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
receive, by the computer, a first video and audio signal front a first teleconference participant;
receive, by the computer, a second video and audio signal from a second teleconference participant;
relay, by the computer, the first video and audio signal to the second teleconference participant and relaying, by the computer, the second video and audio signal to the first teleconference participant;
analyze, by the computer, the first video and audio signal using a plurality of behavior analytics to determine a level of participant engagement of the first teleconference participant, wherein the plurality of behavior analytics includes determining a level of alertness of the first teleconference participant; and
determine, by the computer, that the first teleconference participant is not engaged when the determined level of participant engagement is below a predetermined threshold.

14. The computer program product of claim 13 , wherein the plurality of behavior analytics includes:
determining whether the first teleconference participant is talking to someone or something other than the teleconference;
determining whether the first teleconference participant has turned away from the teleconference;
determining whether the first teleconference participant is engaged in operating, or is receiving a ring from, a telephone unrelated to the teleconference; or
determining an emotional state of the first teleconference participant.

15. The computer program product of claim 13 , wherein when it is determined that the first teleconference participant is not engaged, relaying of the first video and audio signal to the second teleconference

Assignees

• IBM

Inventors

• Dettori Paolo
• Schaffa Frank Andre
• Smilkov Daniel
• Westerink Peter Hans
• Wu Chai Wah

Classifications

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• H04N7/147

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title

• G06K9/00335

  Physics · mapped topic

• G06K9/6269 Primary

  Physics · mapped topic