Method and system of group interaction by user state detection

Patent metadata

Field	Value
Publication number	US-9800834-B2
Application number	US-201615339613-A
Country	US
Kind code	B2
Filing date	Oct 31, 2016
Priority date	Oct 30, 2015
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

A method is disclosed for detecting interaction between two or more participants in a meeting, which includes capturing at least one three-dimensional stream of data on the two or more participants; extracting a time-series of skeletal data from the at least one three-dimensional stream of data on the two or more participants; classifying the time-series of skeletal data for each of the two or more participants based on a plurality of body position classifiers; and calculating an engagement score for each of the two or more participants. In addition, a method is disclosed for improving a group interaction in a meeting, which includes calculating, for each of the two or more participants, an individual engagement state based on attitudes of the participant, wherein the individual engagement state is an engagement state of the participant to the meeting including an engaged state and a disengaged state.

First claim

What is claimed is:

1. A method for detecting interaction between two or more participants in a meeting, the method comprising:
capturing at least one three-dimensional (3D) stream of data on the two or more participants;
extracting a time-series of skeletal data from the at least one 3D stream of data on the two or more participants;
classifying the time-series of skeletal data for each of the two or more participants based on a plurality of body position classifiers;
capturing an audio stream of data on the two or more participants;
classifying the audio stream of data on the two more participants into an utterance classifier based on utterances detected on the audio stream of data, the utterance classifier including at least one of identification of pitch of speech, frequency of speech, and volume of speech;
calculating an engagement score for each of the two or more participants based on the classifying of the time-series of skeletal data and the utterance classifier based on utterances detected on the audio stream for each of the two or more participants; and
providing a feedback in accordance with at least one of the engagement scores of the two or more participants.

2. The method of claim 1 , comprising:
capturing a weight stream of data on the two or more participants, the weight stream of data corresponding to a weight distribution for each of the two or more participants on a chair received from one or more sensors in the chair; and
adding a weight distribution classifier to the engagement score based the weight distribution for each of the two or more participants.

3. The method of claim 1 , comprising:
generating a group engagement score for the two or more participants; and
generating feedback based on the engagement score for each of the two or more participants and the group engagement score, the feedback including one or more of the following:
audio, visual, vibration alert by means of a wearable device, or a change of environmental conditions in a meeting room.

4. The method of claim 1 , comprising:
applying a finite state machine model to the time-series of skeletal data for each of the two or more participants.

5. The method of claim 1 , comprising:
determining trends in meeting between the two or more participants based on the engagement scores for the two or more participants and a group engagement score; and
adjusting meeting times, participants, and/or interaction protocols based on the determined trends.

6. A method for detecting interaction between two or more participants in a meeting, the method comprising:
capturing at least one three-dimensional (3D) stream of data on the two or more participants;
extracting a time-series of skeletal data from the at least one 3D stream of data on the two or more participants;
classifying the time-series of skeletal data for each of the two or more participants based on a plurality of body position classifiers;
calculating an engagement score for each of the two or more participants based on the classifying of the time-series of skeletal data or each of the two or more participants;
providing a feedback in accordance with at least one of the engagement scores of the two or more participants; and
wherein the classifying of the time-series of skeletal data for each of the two or more participants based on the plurality of body position classifiers comprises:
applying a Maximum Mean Discrepancy (MMD) criterion to the time-series of skeletal data to detect change-points over continuous gestures as an initial estimated cuts of gesture transitions;
using kinematic constraints to revise an initial estimated cut to an accurate gesture transition position; and
using a probability density estimation to estimate a hand motion between two cuts to eliminate unintentional movements and non-gesture segments.

7. A non-transitory computer readable medium containing a computer program storing computer readable code for detecting interaction between two or more participant, the program being executable by a computer to cause the computer to perform a process comprising:
capturing at least one three-dimensional (3D) stream of data on the two or more participants;
extracting a time-series of skeletal data from the at least one 3D stream of data on the two or more participants;
classifying the time-series of skeletal data for each of the two or more participants based on a plurality of body position classifiers;
capturing an audio stream of data on the two or more participants;
classifying the audio stream of data on the two more participants into an utterance classifier based on utterances detected on the audio stream of data, the utterance classifier including at least one of identification of pitch of speech, frequency of speech, and volume of speech;
calculating an engagement score for each of the two or more participants based on the classifying of the time-series of skeletal data and the utterance classifier based on utterances detected on the audio stream for each of the two or more participants; and
providing a feedback in accordance with at least one of the engagement scores of the two or more participants.

8. The non-transitory computer readable medium of claim 7 , comprising:
capturing a weight stream of data on a weight distribution of the two or more participants, the weight stream of data corresponding to a weight distribution for each of the two or more participants on a chair received from one or more sensors in the chair; and
adding a weight distribution classifier to the engagement score based the weight distribution for each of the two or more participants.

9. A method for improving a group interaction in a meeting in which two or more participants participate in, the method comprising:
calculating, for each of the two or more participants, an individual engagement state based on attitudes of the participant, wherein the individual engagement state is an engagement state of the participant to the meeting including an engaged state and a disengaged state, wherein the individual engagement state includes:
extracting a time-series of skeletal data from the at least one 3D stream of data on each of the two or more participants;
classifying the time-series of skeletal data for the two or more participants based on a plurality of body position classifiers;
capturing an audio stream of data on the two or more participants;
classifying the audio stream of data on the two more participants into an utterance classifier based on utterances detected on the audio stream of data, the utterance classifier including at least one of identification of pitch of speech, frequency of speech, and volume of speech; and
calculating the individual engagement state for each of the two or more participants based on the classifying of the time-series of skeletal data and the utterance classifier based on utterances detected on the audio stream for each of the two or more participants;

calculating a group engagement state based on the attitudes of the two or more participants, wherein the group engagement state is an engagement state of the two or more participants to the meeting; and
providing a feedback, comprising:
proving a group warning to all of the participants regardless of their individual engagements states, if a number of participants having the disengaged state exceeds a first threshold;
providing a warning only to the participants who have the disengaged state, if the number of the participants who have the disengaged state does not exceed a second threshold; and
providing an environmental feedback to a meeting room system in accordance with the group engagement state.

10. The method as claimed in claim 9 , wh

Assignees

• Konica Minolta Laboratory Usa Inc

Inventors

• Frank Maria
• Tofighi Ghassem
• NAYAK Nandita
• Gu Haisong

Classifications

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• H04N7/15 Primary

  Conference systems · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G08B6/00

  Tactile signalling systems, e.g. tactile personal calling systems · CPC title