Analyzing multimedia content using knowledge graph embeddings

Patent metadata

Field	Value
Publication number	US-10349134-B2
Application number	US-201715639778-A
Country	US
Kind code	B2
Filing date	Jun 30, 2017
Priority date	May 10, 2017
Publication date	Jul 9, 2019
Grant date	Jul 9, 2019

Abstract

A device may receive multimedia data, metadata, and/or policy data. The device may process the policy data using a first set of techniques to determine a first set of embeddings for the policy data. The device may process the multimedia data or the metadata using a second set of techniques to determine a second set of embeddings for the multimedia data or the metadata. The device may process the first set of embeddings and the second set of embeddings using a knowledge graph to determine whether the multimedia content or the access by the user violates the policy. The device may perform an action based on a result of processing the first set of embeddings and the second set of embeddings. The action may relate to the multimedia content or the access to the multimedia content by the user.

First claim

What is claimed is:

1. A device, comprising:
a memory; and
one or more processors to:
receive multimedia data, metadata, and policy data,
the multimedia data being related to multimedia content, and
the policy data being related to a policy that restricts access to particular multimedia content;

process, based on receiving the policy data, the policy data using natural language processing to identify a first set of terms or phrases associated with the policy data;
identify a second set of terms or phrases semantically similar to the first set of terms or phrases;
expand the policy to include the second set of terms or phrases;
determine a first set of embeddings for the first set of terms or phrases and the second set of terms or phrases,
the first set of embeddings identifying the policy data;

determine a set of scores for a set of dimensions associated with an embedding of the first set of embeddings,
the set of scores being based on at least one of:
a context of a particular term or phrase of the first set of terms or phrases or the second set of terms or phrases,
a frequency of the particular term or phrase in the policy data, or
a degree to which other terms or phrases, of the first set of terms or phrase or the second set of terms or phrases, are semantically similar to the particular term or phrase;

process, based on determining the set of scores, the multimedia data and/or the metadata using a set of techniques to determine a second set of embeddings,
the second set of embeddings identifying the multimedia data, and/or
the second set of embeddings identifying the metadata;

process the first set of embeddings and the second set of embeddings using a knowledge graph to determine whether the multimedia content violates the policy; and
perform one or more actions based on a result of processing the first set of embeddings and the second set of embeddings,
the one or more actions including at least one of:
obscuring a portion of the multimedia content,
 the multimedia content being accessible with the portion obscured, or
providing a filtered version of the multimedia content,
 the filtered version of the multimedia content being accessible.

2. The device of claim 1 , where the multimedia data includes:
video data and corresponding audio data; and
where the one or more processors are further to:
align the video data with the corresponding audio data.

3. The device of claim 1 , where the one or more processors are further to:
populate the knowledge graph with information related to the first set of embeddings and the second set of embeddings; and
where the one or more processors, when processing the first set of embeddings and the second set of embeddings, are to:
process the first set of embeddings and the second set of embeddings using the knowledge graph after populating the knowledge graph with the information related to the first set of embeddings and the second set of embeddings.

4. The device of claim 1 , where the one or more actions are one or more first actions;
where the result is a first result; and
where the one or more processors are further to:
perform a second action based on a second result of processing the first set of embeddings and the second set of embeddings,
the second action including:
preventing the access to the multimedia content
permitting the access to the multimedia content, or
providing a recommendation to an administrator related to whether access should be provided to the multimedia content.

5. The device of claim 1 , where the metadata is related to:
a destination to which the multimedia content is to be provided,
a characteristic of a user to whom the multimedia content is to be provided, and/or
a time when the multimedia content is to be provided.

6. A method, comprising:
receiving, by a device, multimedia data, metadata, or policy data,
the multimedia data, the metadata, or the policy data to be used to determine whether multimedia content or access to the multimedia content by a user violates a policy associated with the multimedia content;

processing, by the device and based on receiving the policy data, the policy data using natural language processing to determine a first set of terms or phrases for the policy data;
identifying, by the device, a second set of terms or phrases semantically similar to the first set of terms or phrases;
expanding, by the device, the policy to include the second set of terms or phrases;
determining, by the device, a first set of embeddings for the first set of terms or phrases and the second set of terms or phrases;
determining, by the device, a set of scores for a set of dimensions associated with an embedding of the first set of embeddings,
the set of scores being based on at least one of:
a context of a particular term or phrase of the first set of terms or phrases or the second set of terms or phrases,
a frequency of the particular term or phrase in the policy data, or
a degree to which other terms or phrases, of the first set of terms or phrase or the second set of terms or phrases, are semantically similar to the particular term or phrase;

processing, by the device and based on determining the set of scores, the multimedia data or the metadata using a set of techniques to determine a second set of embeddings for the multimedia data or the metadata;
processing, by the device, the first set of embeddings and the second set of embeddings using a knowledge graph to determine whether the multimedia content or the access by the user violates the policy; and
performing, by the device, one or more actions based on a result of processing the first set of embeddings and the second set of embeddings,
the one or more actions including at least one of:
obscuring a portion of the multimedia content,
the multimedia content being accessible with the portion obscured, or

providing a filtered version of the multimedia content,
the filtered version of the multimedia content being accessible.

7. The method of claim 6 , where processing the multimedia data or the metadata comprises:
processing the multimedia data or the metadata using a technique, of the set of techniques, associated with a type of the multimedia data or the metadata,
the type including at least one of:
audio data,
video data, or
the metadata.

8. The method of claim 6 , where processing the multimedia data or the metadata comprises:
processing audio data to extract a set of audio sequences from the audio data;
processing the set of audio sequences using a recurrent neural network after processing the audio data; and
determining the second set of embeddings for the audio data after processing the set of audio sequences using the recurrent neural network.

9. The method of claim 6 , where processing the multimedia data or the metadata comprises:
processing video data to extract a set of video sequences or frames from the video data; and
processing the set of video sequences or frames using a set of neural networks after processing the video data.

10. The method of claim 6 , where processing the multimedia data or the metadata comprises:
processing video data using a motion extraction model to identify motion shown in the video data.

11. The method of claim 6 , where processing the multimedia data or the metadata comprises:
processing the metadata using a natural language processing technique to identify another set of terms or phrases included in the metadata.

12. A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more

Assignees

• Accenture Global Solutions Ltd

Inventors

• Hamiti Sofian
• Limsopatham Nut
• Zaman Md Faisal
• Lecue Freddy
• Oliveira Antonino Victor
• Kaila Gaurav

Classifications

• H04N21/25841

  involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00) · CPC title

• H04N21/454 Primary

  Content {or additional data} filtering, e.g. blocking advertisements · CPC title

• H04N21/2353

  specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata · CPC title

• H04N21/435

  Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream · CPC title

• G06F17/30029

  Physics · mapped topic