Computerized system and method for automatic highlight detection from live streaming media and rendering within a specialized media player

Patent metadata

Field	Value
Publication number	US-12250415-B2
Application number	US-202217898872-A
Country	US
Kind code	B2
Filing date	Aug 30, 2022
Priority date	Jul 13, 2016
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

Disclosed are systems and methods for improving interactions with and between computers in content generating, searching, hosting and/or providing systems supported by or configured with personal computing devices, servers and/or platforms. The systems interact to identify and retrieve data within or across platforms, which can be used to improve the quality of data used in processing interactions between or among processors in such systems. The disclosed systems and methods automatically analyze a live streaming media file, and identify portions of the media that are highlights. The content classified as a highlight can be shared across social media platforms, and indexed for searching respective to attributes of the video content. The streaming and highlight media content is renderable in a novel, modified video player that enables variable playback speeds for how content is classified, and enables on-demand selections of specific content portions and adjustable rendering displays during streaming.

First claim

What is claimed is:

1. A method comprising:
receiving a streaming media file associated with a live broadcast of an event, the streaming media file comprising a plurality of frames;
determining a set of frames from the plurality of frames corresponding to game scenes;
applying a highlight classifier to each frame of the set of frames to determine a highlight score for each frame as the streaming media file is received;
dynamically adjusting a highlight threshold based on a distribution of highlight scores across the set of frames;
determining a subset of frames of the set of frames with a highlight score above a threshold;
generating a highlight video clip including at least one frame of the subset of frames;
updating a status bar data structure with temporal locations of the highlight video clip within the live broadcast; and
transmitting the highlight video clip and the status bar data structure to a computing device during the live broadcast.

2. The method of claim 1 , wherein a game scene is a scene type and wherein determining the set of frames comprises labeling, by a scene classifier based on a stored index file, each frame of the set of frames as a game scene, wherein the stored index file includes information related to previously determined labels and scene types.

3. The method of claim 2 , further comprising generating the stored index file, wherein generating the stored index file comprises:
receiving a set of training videos;
identifying a set of training frames from the set of training videos;
determining one or more labels for the set of training frames based on a type of content within the set of training frames; and
generating the stored index file based on the one or more labels and the type of content.

4. The method of claim 2 , further comprising generating the stored index file, wherein generating the stored index file comprises:
receiving a log file comprising activity information corresponding to a set of training videos;
analyzing the log file to identify one or more scene types based on the activity information;
labeling the set of training videos based on the identified one or more scene types with one or more labels; and
generating the stored index file based on the activity information and the one or more labels.

5. The method of claim 1 , wherein the highlight classifier determines the highlight score based on a content of the frame.

6. The method of claim 1 , further comprising automatically communicating, over a network, upon generation of said highlight video clip, the highlight video clip to a user.

7. The method of claim 1 , wherein the highlight classifier includes a Convolutional Neural Network having a last layer including a softmax function.

8. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a device, cause the device to:
receive a streaming media file associated with a live broadcast of an event, the streaming media file comprising a plurality of frames;
determine a set of frames from the plurality of frames corresponding to game scenes;
apply a highlight classifier to each frame of the set of frames to determine a highlight score for each frame as the streaming media file is received;
dynamically adjust a highlight threshold based on a distribution of highlight scores across the set of frames;
determine a subset of frames of the set of frames with a highlight score above a threshold;
generate a highlight video clip including at least one frame of the subset of frames;
update a status bar data structure with temporal locations of the highlight video clip within the live broadcast; and
transmit the highlight video clip and the status bar data structure to a computing device during the live broadcast.

9. The computer-readable storage medium of claim 8 , wherein a game scene is a scene type and wherein determining the set of frames comprises labeling, by a scene classifier based on a stored index file, each frame of the set of frames as a game scene, wherein the stored index file includes information related to previously determined labels and scene types.

10. The computer-readable storage medium of claim 9 , wherein the instructions further configure the device to generate the stored index file, wherein generating the stored index file comprises:
receiving a set of training videos;
identifying a set of training frames from the set of training videos;
determining one or more labels for the set of training frames based on a type of content within the set of training frames; and
generating the stored index file based on the one or more labels and the type of content.

11. The computer-readable storage medium of claim 9 , wherein the instructions further configure the device to generate the stored index file, wherein generating the stored index file comprises:
receiving a log file comprising activity information corresponding to a set of training videos;
analyzing the log file to identify one or more scene types based on the activity information;
labeling the set of training videos based on the identified one or more scene types with one or more labels; and
generating the stored index file based on the activity information and the one or more labels.

12. The computer-readable storage medium of claim 8 , wherein the highlight classifier determines the highlight score based on a content of the frame.

13. The computer-readable storage medium of claim 8 , wherein the instructions further configure the device to automatically communicate, over a network, upon generation of said highlight video clip, the highlight video clip to a user.

14. The computer-readable storage medium of claim 8 , wherein the highlight classifier includes a Convolutional Neural Network having a last layer including a softmax function.

15. A device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the device to:
receive a streaming media file associated with a live broadcast of an event, the streaming media file comprising a plurality of frames;
determine a set of frames from the plurality of frames corresponding to game scenes;
apply a highlight classifier to each frame of the set of frames to determine a highlight score for each frame as the streaming media file is received;
dynamically adjust a highlight threshold based on a distribution of highlight scores across the set of frames;
determine a subset of frames of the set of frames with a highlight score above a threshold;
generate a highlight video clip including at least one frame of the subset of frames;
update a status bar data structure with temporal locations of the highlight video clip within the live broadcast; and
transmit the highlight video clip and the status bar data structure to a computing device during the live broadcast.

16. The device of claim 15 , wherein a game scene is a scene type and wherein determining the set of frames comprises labeling, by a scene classifier based on a stored index file, each frame of the set of frames as a game scene, wherein the stored index file includes information related to previously determined labels and scene types.

17. The device of claim 16 , wherein the instructions further configure the device to generate the stored index file, wherein generating the stored index file comprises:
receiving a set of training videos;
identifying a set of training frames from the set of training videos;
determining one or more labels for the set of training frames based on a type of co

Assignees

• Yahoo Assets Llc

Inventors

• Ni Bin
• Lieb Kirk
• Hawes Rick
• Song Yale
• Schillings Benoit
• Oughourlian Vahe
• Vallmitjana Jordi
• Nystrom Jennelle
• Ruparel Hardik
• Chen Michael
• Mathes Adam
• Balasubramanian Arunkumar
• Zhou Jian
• Edelman Matt

Classifications

• G11B27/28

  by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04N21/8549

  Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} · CPC title

• H04N21/2387

  Stream processing in response to a playback request from an end-user, e.g. for trick-play · CPC title

• H04N21/4882

  for displaying messages, e.g. warnings, reminders (arrangements for providing short real-time information to substation in data switching networks H04L12/1895) · CPC title