System and method for improving communication productivity

Patent metadata

Field	Value
Publication number	US-9639770-B2
Application number	US-201514669212-A
Country	US
Kind code	B2
Filing date	Mar 26, 2015
Priority date	Mar 26, 2015
Publication date	May 2, 2017
Grant date	May 2, 2017

Abstract

A method, computer readable storage medium, and system are disclosed for improving communication productivity, comprising: capturing at least one three-dimensional (3D) stream of data on two or more subjects; extracting a time-series of skeletal data from the at least one 3D stream of data on the two or more subjects; and determining an engagement index between the two or more subjects by comparing the time-series of skeletal data on each of the two or more subjects over a time window.

First claim

What is claimed is:

1. A method for improving communication productivity, comprising:
capturing at least one three-dimensional (3D) stream of data on each of two or more subjects, wherein at least one of the two or more subjects is classified as a speaker and one or more of the two or more subjects is classified as a listener;
extracting a time-series of skeletal data from the at least one 3D stream of data on each of the two or more subjects;
generating a synchrony score for the two or more subjects, wherein the synchrony score is calculated by comparing the time-series of skeletal data of each of the two or more subjects to one another for a defined period of time; and
using the synchrony score to generate an engagement index between the two or more subjects.

2. The method of claim 1 , comprising:
providing feedback in real time to at least one of the two or more subjects if the engagement index between the two or more subjects is classified as a low interaction.

3. The method of claim 1 , comprising:
classifying the at least one of the two or more subjects as the speaker and the one or more of the two or more subjects as the listener based on video and audio correlation of the stream of data.

4. The method of claim 1 , comprising;
generating the synchrony score for each of the two or more subjects at a same time of a conversation.

5. The method of claim 1 , comprising:
generating a synchrony score for the one or more of the two or more subjects classified as the listener by comparing skeletal data at a starting point of a conversation and at one or more subsequent points of the conversation.

6. The method of claim 1 , wherein the the synchrony score between the two or more subjects is generated by comparing one or more of the following features:
head and torso;
right arm;
left arm;
right leg; and
left leg.

7. The method of claim 1 , further comprising:
detecting facial expressions or gestures for each of the two or more subjects;
comparing the facial expressions or gestures; and
combining the comparison of the facial expressions or gestures and the comparison of the series of skeletal data on each of the two or more subjects to generate the engagement index score for each of the two or more subjects.

8. The method of claim 1 , wherein the synchrony score is calculated using the following:
a mean, which represents average angle of a joint;
a standard deviation, which measures an amount an angle varies over time; and
an estimated-skewness, which represents a measure of how much the angle deviates from the mean.

9. A non-transitory computer readable medium containing a computer program storing computer readable code for improving communication productivity, the program being executable by a computer to cause the computer to perform a process comprising:
capturing at least one three-dimensional (3D) stream of data on each of two or more subjects, wherein at least one of the two or more subjects is classified as a speaker and one or more of the two or more subjects is classified as a listener;
extracting a time-series of skeletal data from the at least one 3D stream of data on each of the two or more subjects;
generating a synchrony score for the two or more subjects, wherein the synchrony score is calculated by comparing the skeletal data of each of the two or more subjects to one another for a defined period of time; and
using the synchrony score to generate an engagement index between the two or more subjects.

10. The computer readable storage medium of claim 9 , comprising
providing feedback in real time to at least one of the two or more subjects if the engagement index between the two or more subjects is classified as a low interaction.

11. The computer readable storage medium of claim 9 , comprising:
classifying at least one of the one of the two or more subjects as the speaker and the one or more of the two or more subjects as the listener based on video and audio correlation of the 3D stream of data; and
generating the synchrony score for each of the two or more subjects at a same time of a conversation.

12. The computer readable storage medium of claim 9 , comprising:
generating a synchrony score for the one or more of the two or more subjects classified as the listener by comparing skeletal data at a starting point of a conversation and at one or more subsequent points of the conversation.

13. The computer readable storage medium of claim 9 , wherein the synchrony score between the two or more subjects is generated by comparing one or more of the following features:
head and torso;
right arm;
left arm;
right leg; and
left leg.

14. The computer readable storage medium of claim 9 , further comprising:
detecting facial expressions or gestures for each of the two or more subjects;
comparing the facial expressions or gestures; and
combining the comparison of the facial expressions or gestures and the comparison of the series of skeletal data on each of the two or more subjects to generate the engagement index score for each of the two or more subjects.

15. A system for improving communication productivity, comprising:
a video camera for capturing at least one three-dimensional (3D) stream of data on each of two or more subjects; and
a processor configured to:
extract a time-series of skeletal data from the at least one 3D stream of data on each of the two or more subjects,
classify at least one of the two or more subjects as a speaker and one or more of the two or more subjects as a listener;
generate a synchrony score for the two or more subjects, wherein the synchrony score is calculated by comparing the skeletal data of each of the two or more subjects to one another for a defined period of time; and

use the synchrony score to generate an engagement index between the two or more subjects.

16. The system of claim 15 , comprising:
a speaker, a display, or a vibration alert wearable, which is configured to provide feedback in real time to at least one of the two or more subjects if the engagement index between the two or more subjects is classified as a low interaction.

17. The system of claim 15 , wherein the processor is configured to:
classify the at least one of the one of the two or more subjects as the speaker and the one or more of the two or more subjects as the listener based on video and audio correlation of the 3D stream of data.

18. The system of claim 15 , wherein the processor is configured to:
generate the synchrony score for each of the two or more subjects at a same time of a conversation.

19. The system of claim 15 , wherein the processor is configured to:
generate a synchrony score for the one or more of the two or more subjects classified as the listener by comparing skeletal data at a starting point of a conversation and at one or more subsequent points of the conversation.

20. The system of claim 15 , wherein the synchrony score between the two or more subjects is generated by comparing one or more of the following features:
head and torso;
right arm;
left arm;
right leg; and
left leg.

Assignees

• Konica Minolta Laboratory Usa Inc

Inventors

• Paradkar Foram Manish
• Zhang Yongmian
• Gu Haisong

Classifications

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V10/34

  Smoothing or thinning of the pattern; Morphological operations; Skeletonisation · CPC title

• G06V10/426

  Graphical representations · CPC title

• G06F17/30516

  Physics · mapped topic