System and method for real-time, event-driven video conference analytics

Patent metadata

Field	Value
Publication number	US-11937016-B2
Application number	US-202117331154-A
Country	US
Kind code	B2
Filing date	May 26, 2021
Priority date	May 26, 2021
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A system, platform, computer program product, and/or method to analyze a computer-implemented video conference includes: a plurality of participant devices, and a central processing server. Each participant device is configured to form a video snippet for a time interval of the video conference having audio data and video data; generate a transformed video snippet by embedding extracted participant data and/or metadata into the video snippet; and send each transformed video snippet to the central processing server. The central processing server receives each transformed video snippet; performs analytics on each transformed video snippet; and transmits to at least one of the participant devices, results of the performed analytics. Participant devices can display one or more results of the performed analytics.

First claim

What is claimed is:

1. A computer-implement method of analyzing a computer implemented video conference that includes a plurality of participant devices, the method comprising:
receiving, by a central processing server having a processor, one or more transformed video snippets sent by at least one of the plurality of participant devices during the video conference, wherein the one or more transformed video snippets comprise audio and video data collected from the at least one of the plurality of participant devices at time intervals in the form of video snippets embedded with at least one of an extracted group consisting of: extracted participant data, extracted participant metadata, and combinations thereof;
using, by the central processing server, a video conference identifier embedded in the one or more transformed video snippets to match the one or more transformed video snippets, sent by the at least one of the plurality of participant devices to the central processing server, to a specific video conference; and
performing analytics, by the central processing server, on the one or more transformed video snippets received by the central processing server for a current time interval.

2. The computer-implemented method recited in claim 1 , further comprising:
transferring, from the central processor to the at least one of the plurality of participant devices, a copy of an event handler, wherein the copy of the event handler comprises programming instructions, which when executed by the at least one of the plurality of participant devices, configures the at least one of the plurality of participant devices to:
form, by the at least one of the plurality of participant devices, a plurality of video snippets at time intervals during the video conference, each of the plurality of video snippets comprising audio and video data collected during a time interval of the video conference from the at least one of the plurality of participant devices;
extract from the at least one of the plurality of participant devices during the video conference one of an extracted group consisting of: participant data; participant metadata;

and combinations thereof;
generate a transformed video snippet by the at least one of the plurality of participant devices at time intervals, by embedding into a video snippet using video stenography the at least one of the extracted group; and
send by the at least one of the plurality of participant devices during the video conference, after each time interval, the transformed video snippet formed by the at least one of the plurality of participant devices to the central processing server.

3. The computer implemented method recited in claim 1 , wherein performing analytics on the one or more transformed video snippets received by the central processing server for a current time interval comprises at least one of the group consisting of:
using a tone analyzer to determine the tone of the audio in one or more transformed video snippets received by the central processing server for the current time interval;
using natural language processing to analyze text extracted from the audio in the one or more transformed video snippets received by the central processing server for the current time interval;
using machine learning algorithms to analyze at least one of the group consisting of the participant data, the participant metadata, and combinations thereof from the one or more transformed video snippets received by the central processing server for the current time interval;
using at least one of the group consisting of the participant data, the participant metadata, and combinations thereof embedded in the one or more transformed video snippets received by the central processing server for the current time period to determine at least one of the group consisting of: background noise level of each participant device; location of each participant device; whether a participant device collected audio data; whether a participant device shared its display; each participant device condition information; each participant device network latency; whether a participant device had its audio capture device muted; whether a participant device logged onto or off the video conference, and combinations thereof; and
combinations thereof.

4. The computer-implemented method recited in claim 1 , further comprising transmitting one or more results of the performed analytics to one or more participant devices participating in the video conference.

5. The computer-implemented method recited in claim 1 , further comprising:
combining, by the central processing server, the analytics performed on the one or more transformed video snippets received by the central processing server for the current time interval with the performed analytics for all subsequent transformed video snippets received by the central processing server for all prior time intervals.

6. The computer-implemented method recited in claim 5 , further comprising transmitting one or more results of the combined analytics to any one of the plurality of participant devices.

7. The computer-implemented method recited in claim 1 , wherein the time intervals are at least one of the group consisting of: fixed, predetermined, preset, adjustable, configurable, programable, machine learned, and combinations thereof.

8. The computer-implemented method recited in claim 7 , wherein each time interval is a fixed time period, wherein the time period ranges from 2 seconds to 20 seconds.

9. A computer-implemented method for analyzing a computer-implemented video conference including a plurality of participant devices, each participant device having a processor and the ability to capture audio data and video data, the method comprising:
forming, by a participant device, a video snippet for a first time interval of the video conference, the video snippet comprising audio data and video data collected during the first time interval from the participant device;
extracting, by the participant device, at least one of an extracted group consisting of: participant data, participant metadata, and combinations thereof from the participant device;
generating, by the participant device, a transformed video snippet by embedding into the video snippet, by the participant device, the at least one of the extracted group;
approving collection and use of, during the video conference by the participant device, the at least one of the extracted group; and
sending, by the participant device during the video conference, after the first time interval, the transformed video snippet generated by the participant device to a central processing server.

10. The computer-implemented method according to claim 9 , wherein forming, by the participant device, a transformed video snippet comprises embedding into the video snippet using video stenography the at least one of the extracted group.

11. The computer-implemented method according to claim 9 , wherein the first time interval is at least one of the group consisting of fixed, predetermined, preset, adjustable, configurable, programable, machine learned, and combinations thereof.

12. The computer-implemented method according to claim 9 , further comprising providing an event handler on each participant device wherein the event handler comprises programming instructions that when executed by the participant device cause the participant device to perform the forming, extracting, generating, and sending steps of claim 9 .

13. The computer-implemented method according to claim 9 , wherein the at least one of the extracted group during the first time interval consists of at least one of the follo

Assignees

• IBM

Inventors

• Oluyemi Oluwanifemi
• Movva Vamsee
• Ran Chuan
• Sharif A Jaylani

Classifications

• H04N7/155 Primary

  involving storage of or access to video conference sessions (tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831) · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06N20/00

  Machine learning · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title