Generating moving thumbnails for videos

Patent metadata

Field	Value
Publication number	US-10347294-B2
Application number	US-201615199269-A
Country	US
Kind code	B2
Filing date	Jun 30, 2016
Priority date	Jun 30, 2016
Publication date	Jul 9, 2019
Grant date	Jul 9, 2019

Abstract

A method of generating a moving thumbnail is disclosed. The method includes sampling video frames of a video item. The method further includes determining frame-level quality scores for the sampled video frames. The method also includes determining multiple group-level quality scores for multiple groups of the sampled video frames using the frame-level quality scores of the sampled video frames. The method further includes selecting one of the groups of the sampled video frames based on the multiple group-level quality scores. The method includes creating a moving thumbnail using a subset of the video frames that have timestamps within a range from the start timestamp to the end timestamp.

First claim

We claim:

1. A method comprising:
sampling a plurality of video frames of a video item;
determining frame-level quality scores for the sampled video frames of the video item, wherein a frame-level quality score of a respective one of the sampled video frames is indicative of a quality of the respective video frame of the video item;
applying a sliding window to the sampled video frames to identify a plurality of groups of the sampled video frames for scoring on a group-level, wherein the sliding window groups multiple sets of consecutively sampled video frames into the plurality of groups of the sampled video frames, wherein at least two of the multiple sets of consecutively sampled video frames overlap, comprise a same sampled video frame and are scored on the group-level to provide an indication of a quality of a respective set of consecutively sampled video frames for inclusion in a moving thumbnail;
determining, by a processing device, a plurality of group-level quality scores for the plurality of groups of the sampled video frames of the video item, the plurality of groups identified by applying the sliding window, wherein the group-level quality scores are determined using the frame-level quality scores of the sampled video frames;
selecting one of the plurality of groups of the sampled video frames of the video item based on the plurality of group-level quality scores, wherein the group of sampled video frames selected based on a respective group level quality score includes a set of consecutively sampled video frames of the multiple sets of consecutively sampled video frames and has a start timestamp associated with a first sampled video frame in the set of consecutively sampled video frames and an end timestamp associated with a last sampled video frame in the set of consecutively sampled video frames; and
creating the moving thumbnail using the group of sampled video frames selected based on the respective group level quality score, the moving thumbnail comprising a subset of the plurality of video frames that have timestamps within a range from the start timestamp to the end timestamp.

2. The method of claim 1 , wherein the plurality of video frames of the video item are sampled at a fixed sampling rate.

3. The method of claim 1 , wherein the plurality of video frames of the video item are sampled from a beginning portion of the video item.

4. The method of claim 1 , wherein determining the frame-level quality scores for the sampled video frames comprises:
generating one or more quality scores for each of the sampled video frames in view of one or more of a plurality of frame-level scorers; and
summing the one or more quality scores for each of the sampled video frames to generate the frame-level quality scores for the sampled video frames.

5. The method of claim 4 , generating one or more quality scores for each of the sampled video frames in view of one or more of a plurality of frame-level scorers comprises:
generating a first quality score for each of the sampled video frames using a thumbnail quality scorer that measures a visual quality of the sampled video frames;
generating a second quality score for each of the sampled video frames using a face scorer that measures a size of a face relative to a size of the sampled video frames; and
generating a third quality score for each of the sampled video frames using a motion scorer that measures motion between two consecutively sampled video frames.

6. The method of claim 4 , wherein summing the quality scores for each the sampled video frames comprises summing the quality scores using a linear weighted summation.

7. The method of claim 1 , wherein applying the sliding window to the sampled video frames to identify the plurality of groups of the sampled video frames for scoring on the group-level further comprises:
applying the sliding window to different positions along a time axis to group the sampled video frames into the plurality of groups of the sampled video frames, wherein the sampled video frames are ordered with respect to the time axis, and wherein frame-level quality scores of a particular group of sampled video frames are used to determine a group-level quality score for the particular group.

8. The method of claim 1 , wherein applying the sliding window to the sampled video frames to identify the plurality of groups of the sampled video frames for scoring on the group-level further comprises:
applying the sliding window to a first position with respect to a time axis, wherein the sliding window extends for a fixed duration and a first group of the plurality of groups of the sampled video frames is within the sliding window at the first position with respect to the time axis; and
applying the sliding window to a second position with respect to the time axis, wherein a second group of the plurality of groups of the sampled video frames is within the sliding window at the second position with respect to the time axis; and
wherein determining the plurality of group-level quality scores for the plurality of groups of the sampled video frames using the frame-level quality scores of the sampled video frames further comprises:
determining a first group-level quality score of the plurality of group-level quality scores by aggregating frame-level quality scores associated with the first group of sampled video frames; and
determining a second group-level quality score of the plurality of group-level quality scores by aggregating frame-level quality scores associated with the second group of sampled video frames.

9. The method of claim 1 , wherein the selected group has a highest group-level quality score.

10. The method of claim 1 , further comprising
modifying the subset of the plurality of video frames of the moving thumbnail to cause at least one of a first video frame of the subset of the plurality of video frames or a last video frame of the subset of the plurality of video frames to be proximate a shot boundary of the video item.

11. The method of claim 1 , further comprising
modifying a playback speed of the moving thumbnail in view of motion measured in the subset of the plurality of video frames of the moving thumbnail.

12. The method of claim 1 , wherein the moving thumbnail is in a Graphic Interchange Format (GIF).

13. The method of claim 1 , wherein the moving thumbnail is in an H.264 format.

14. A non-transitory computer-readable medium storing instructions that, responsive to execution by a processing device, cause the processing device to perform operations comprising:
sampling a plurality of video frames of a video item;
determining frame-level quality scores for the sampled video frames of the video item, wherein a frame-level quality score of a respective one of the sampled video frames is indicative of a quality of the respective video frame of the video item;
applying a sliding window to the sampled video frames to identify a plurality of groups of the sampled video frames for scoring on a group-level, wherein the sliding window groups multiple sets of consecutively sampled video frames into the plurality of groups of the sampled video frames, wherein at least two of the multiple sets of consecutively sampled video frames overlap, comprise a same sampled video frame and are scored on the group-level to provide an indication of a quality of a respective set of consecutively sampled video frames for inclusion in a moving thumbnail;
determining, by the processing device, a plurality of group-level quality scores for the plurality of groups of the sampled video frames of the video item, the plurality of groups identified by the sliding

Assignees

• Google Inc
• Google Llc

Inventors

• Yang Weilong
• Tsai Min-Hsuan
• Sun Zheng
• Cao Pei
• Izo Tomas

Classifications

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• G06F16/739

  in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G11B27/326

  used signal is a video-frame or a video-field (P.I.P.) · CPC title