Apparatus, systems, and methods for integrating digital media content

Patent metadata

Field	Value
Publication number	US-10522186-B2
Application number	US-201816049690-A
Country	US
Kind code	B2
Filing date	Jul 30, 2018
Priority date	Jul 28, 2017
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

Disclosed herein are techniques for digital content integration. A computer-implemented method includes receiving a target digital content item that includes a plurality of frames, identifying a set of candidate host frames for inserting source digital content items from the plurality of frames based on one or more attributes of the target digital content item, determining a candidate score for each respective candidate host frame of the candidate host frames, and generating host time defining data including identifications and the candidate scores of the candidate host frames, where the candidate score indicates a degree of transition of the target digital content item at the candidate host frame. One or more candidate host frames are then selected based on the candidate scores for inserting one or more source digital content items into the target digital content item.

First claim

The invention claimed is:

1. A computer-implemented method, comprising, by one or more processing devices:
receiving a target digital content item, the target digital content item comprising a plurality of frames;
identifying, from the plurality of frames and based on one or more attributes of the target digital content item, a set of candidate host frames for inserting source digital content items;
determining a candidate score for each respective candidate host frame of the candidate host frames, wherein the candidate score indicates a degree of transition of the target digital content item at the candidate host frame; and
generating host time defining data including identifications and the candidate scores of the candidate host frames.

2. The method of claim 1 , wherein the one or more attributes of the target digital content item include at least one of metadata associated with the target digital content item, pixel values or objects in visual content of the target digital content item, or a transcript, amplitude, zero crossing rate, energy, entropy of energy, spectral centroids, spectral spread, spectral entropy, spectral flux, spectral roll-off, MFCCs, chroma vector, or chroma deviation of audio content of the target digital content item.

3. The method of claim 1 , further comprising:
ranking the candidate host frames identified in the host time defining data based on their respective candidate scores in the host time defining data;
selecting, based on the ranking, one or more candidate host frames for inserting one or more source digital content items into the target digital content item; and
inserting the one or more source digital content items into the target digital content item at the one or more candidate host frames.

4. The method of claim 1 , wherein:
identifying the set of candidate host frames comprises, for each pair of consecutive frames in the plurality of frames:
determining a level of change of the one or more attributes between the pair of consecutive frames; and
selecting a later frame in the pair of consecutive frames as a candidate host frame based on the level of change being greater than a threshold value; and

determining the candidate score for each candidate host frame comprises determining the candidate score for the candidate host frame based on the level of change of the one or more attributes between the candidate host frame and a frame immediately before the candidate host frame.

5. The method of claim 1 , wherein:
identifying the set of candidate host frames comprises, for each frame in an analyzing interval that includes a set of frames:
determining an attribute of the frame;
determining, as a threshold, a variance of the attribute among the frame and frames before the frame in the analyzing interval; and
identifying the frame as a candidate host frame based on the attribute of the frame being greater than the threshold.

6. The method of claim 1 , wherein:
the one or more attributes comprise pixel values of the target digital content;
identifying the set of candidate host frames comprises, for each frame in the plurality of frames:
determining a level of change of the pixel values between the frame and a white frame or between the frame and a black frame; and
selecting the frame as a candidate host frame based on the level of change being greater than a threshold value; and

determining the candidate score for each candidate host frame comprises determining the candidate score for the candidate host frame based on the level of change of the pixel values between the frame and the white frame or between the frame and the black frame.

7. The method of claim 1 , further comprising:
extracting, using one or more neural networks, a first set of one or more feature vectors from a frame in an analyzing interval before a candidate host frame;
combining the first set of one or more feature vectors into a first content vector;
extracting, using the one or more neural networks, a second set of one or more feature vectors from a frame in an analyzing interval after the candidate host frame;
combining the second set of one or more feature vectors into a second content vector;
determining a distance between the first content vector and the second content vector;
increasing the candidate score of the candidate host frame based on the distance being greater than a first threshold value or removing the candidate host frame from the set of candidate host frames based on the distance being less than a second threshold value.

8. The method of claim 1 , wherein identifying the set of candidate host frames comprises, for each frame in the plurality of frames:
recognizing, using a neural network, objects in the frame;
comparing the objects recognized in the frame with objects recognized in an earlier frame;
identifying the frame as a candidate host frame based on the objects recognized in the frame different from the objects recognized in the earlier frame.

9. The method of claim 1 , further comprising:
determining, for each of the candidate host frames, a difference of the one or more attributes between a frame immediately preceding the candidate host frame and a frame immediately following the candidate host frame; and
reducing the candidate score of the candidate host frame or removing the candidate host frame from the set of candidate host frames based on the difference being less than a threshold.

10. The method of claim 1 , further comprising, for each candidate host frame in the candidate host frames:
determining, for a first analyzing interval immediately preceding the candidate host frame, a first motion vector for frames in the first analyzing interval using an optical flow method;
determining, for a second analyzing interval immediately following the candidate host frame, a second motion vector for frames in the second analyzing interval using the optical flow method;
determining a difference between the first motion vector and the second motion vector; and
decreasing the candidate score of the candidate host frame or removing the candidate host frame from the set of candidate host frames based on the difference being less than a threshold.

11. The method of claim 1 , wherein:
the target digital content item comprises audio data;
identifying the set of candidate host frames comprises:
classifying frames or segments of the audio data, the classifying including determining a degree of confidence of the classification for each respective frame or segment of the audio data; and
identifying, as a candidate host frame, a frame immediately before a change of classification among the frames or segments of the audio data; and

determining the candidate score for the candidate host frame comprises determining the candidate score for the candidate host frame based on the degree of confidence associated with the candidate host frame.

12. The method of claim 1 , wherein:
the target digital content item comprises audio data;
identifying the set of candidate host frames comprises:
classifying segments of the audio data;
identifying two or more segments of the audio data classified as human speech based on the classification; and
determining a change of speakers among the two or more segments based on an amplitude or Mel-Frequency Cepstral Coefficients (MFCCs) associated with each of the two or more segments, wherein determining the change of speakers includes determining a degree of confidence of determining the change of speakers; and
identifying, as a candidate host frame, a frame immediately before the change of speakers; and

determining the candidate score for the candidate host frame compri

Assignees

• Adobe Inc

Inventors

• Attorre Brunno Fidel Maciel
• Xue Xiaozhen
• Marzban Shabbir
• Thien Nicolas Huynh
• Marino William L

Classifications

• G10L21/007

  characterised by the process used · CPC title

• G10L25/09

  the extracted parameters being zero crossing rates · CPC title

• H04N21/233

  Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• G10L25/24

  the extracted parameters being the cepstrum · CPC title