Automatic video segment selection method and apparatus

Patent metadata

Field	Value
Publication number	US-9911223-B2
Application number	US-201615154038-A
Country	US
Kind code	B2
Filing date	May 13, 2016
Priority date	May 13, 2016
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Disclosed are systems and methods for improving interactions with and between computers in content generating, searching, hosting and/or providing systems supported by or configured with personal computing devices, servers and/or platforms. The systems interact to identify and retrieve data within or across platforms, which can be used to improve the quality of data used in processing interactions between or among processors in such systems. The disclosed systems and methods provide systems and methods for automatically extracting and creating an animated Graphics Interchange Format (GIF) file from a media file. The disclosed systems and methods identify a number of GIF candidates from a video file, and based on analysis of each candidate's attributes, features and/or qualities, at least one GIF candidate is automatically selected for rendering.

First claim

The invention claimed is:

1. A method comprising:
receiving, at a computing device, a request from a user for creation of an animated Graphics Interchange Format (GIF) file from a video file;
determining, via the computing device, a plurality of segments within the video file, the segment determination comprising parsing the video file to identify transition frames within the video file, each segment of the plurality comprising video frames of the video file existing between a pair of identified transition frames;
determining, via the computing device, an n-dimensional feature vector for each segment of the plurality, the feature vector determination for the segment of the plurality comprising parsing the segment to identify information associated with context and content features of the segment, the feature vector based on the context and content information;
determining, via the computing device, a GIF suitability score for each segment of the plurality, the GIF suitability score determination for the segment of the plurality being based on the segment's feature vector, the segment's GIF suitability score being a measure of the segment's suitability for inclusion in the animated GIF file;
automatically selecting, via the computing device, a predetermined number of segments of the plurality for inclusion in the animated GIF file based on the GIF suitability score of each segment of the plurality, each selected segment's GIF suitability score indicating that the selected segment is more suitable for inclusion in the animated GIF file than each unselected segment's GIF suitability score; and
automatically creating, via the computing device, the animated GIF file, the animated GIF file comprising each selected segment.

2. The method of claim 1 , further comprising:
communicating, via the computing device, the animated GIF file to a user for display on a device of the user.

3. The method of claim 1 , wherein the parsing of the segment to identify the transition frames comprises applying a frame differencing algorithm.

4. The method of claim 1 , wherein the transition frames comprise a visual effect selected from a group consisting of: a cut between video frames, fade in/out between frames, dissolve and wipe.

5. The method of claim 1 , further comprising:
determining the context information by extracting information from metadata associated with the video file, the metadata comprising a category label indicating a content category of the video file represented using a one hot vector representation; and
determining the content information by analyzing the segment's content using a three-dimensional convolutional neural network comprising a number of spatio-temporal convolutional layers capturing both spatial and temporal features of the segment.

6. The method of claim 5 , the context information determination further comprising extracting at least one of a title, descriptive tags and user commentary, each of which is represented using a semantic embedding.

7. The method of claim 5 , at least a portion of the context information for a segment being extracted from metadata is associated with the segment and a remaining portion of the content information being associated with the video file.

8. The method of claim 5 , the feature vector determination further comprising:
combining, for each segment of the plurality, the segment's context and content information into the segment's n-dimensional feature vector.

9. The method of claim 1 , the GIF suitability score determination further comprising:
applying a regressive ranking function to the feature vector in order to determine the GIF suitability score, the application of the regressive ranking function is based on a training dataset of known GIFs and a plurality of video files from which the known GIFs were generated, the regression function being trained using a plurality of segment pairs, each segment pair comprising a pair of segments from a same video file of the plurality, a first segment of the segment pair being included in a known GIF and a second segment of the pair from the same video file being excluded from the known GIF, the application of the regressive ranking function is further based on a loss function determining whether to impose a penalty to ensure that the first segment is scored higher than the second segment by a ranking margin.

10. The method of claim 9 , the penalty determined by the rank loss function being nonzero unless the first segment is ranked higher than the second segment by a certain margin, the nonzero penalty being a quadratic penalty for a small violation of the ranking margin and a linear penalty for other than the small violation, the small violation of the ranking margin being determined using a margin violation threshold.

11. The method of claim 10 , the margin violation threshold varies based on GIF popularity, such that the margin violation threshold used for a popular known GIF is higher than the margin violation threshold used for a less popular known GIF.

12. The method of claim 1 , the automatic selection of a predetermined number of segments comprising automatically selecting the predetermined number of top-ranked segments of the plurality, the plurality of segments being ranked using each segment's GIF suitability score.

13. The method of claim 1 , each selected segment's GIF suitability score satisfies a GIF suitability score threshold.

14. The method of claim 1 , the predetermined number of segments corresponds to a number of segments being requested.

15. The method of claim 1 , further comprising:
receiving, at the computing device, a user request for an animated GIF; and
identifying, via the computing device and from a set of animated GIFs each of which having a corresponding GIF suitability score, a number of animated GIFs using each animated GIF's corresponding GIF suitability score as a measure of quality of the animated GIF, the identifying comprising selecting, from the set, the number of animated GIFs having the highest GIF suitability score relative to each unselected animated GIF in the set.

16. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a computing device, performs a method comprising:
receiving a request from a user for creation of an animated Graphics Interchange Format (GIF) file from a video file;
determining a plurality of segments within the video file, the segment determination comprising parsing the video file to identify transition frames within the video file, each segment of the plurality comprising video frames of the video file existing between a pair of identified transition frames;
determining an n-dimensional feature vector for each segment of the plurality, the feature vector determination for the segment of the plurality comprising parsing the segment to identify information associated with context and content features of the segment, the feature vector based on the context and content information;
determining a GIF suitability score for each segment of the plurality, the GIF suitability score determination for the segment of the plurality being based on the segment's feature vector, the segment's GIF suitability score being a measure of the segment's suitability for inclusion in the animated GIF file;
automatically selecting a predetermined number of segments of the plurality for inclusion in the animated GIF file based on the GIF suitability score of each segment of the plurality, each selected segment's GIF suitability score indicating that the selecte

Assignees

• Yahoo Holdings Inc

Inventors

• Song Yale
• Gygli Michael
• Cao Liangliang

Classifications

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06T13/80 Primary

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title