Computerized system and method for automatic highlight detection from live streaming media and rendering within a specialized media player

Patent metadata

Field	Value
Publication number	US-11438637-B2
Application number	US-202015930726-A
Country	US
Kind code	B2
Filing date	May 13, 2020
Priority date	Jul 13, 2016
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

Disclosed are systems and methods for improving interactions with and between computers in content generating, searching, hosting and/or providing systems supported by or configured with personal computing devices, servers and/or platforms. The systems interact to identify and retrieve data within or across platforms, which can be used to improve the quality of data used in processing interactions between or among processors in such systems. The disclosed systems and methods automatically analyze a live streaming media file, and identify portions of the media that are highlights. The content classified as a highlight can be shared across social media platforms, and indexed for searching respective to attributes of the video content. The streaming and highlight media content is renderable in a novel, modified video player that enables variable playback speeds for how content is classified, and enables on-demand selections of specific content portions and adjustable rendering displays during streaming.

First claim

What is claimed is:

1. A computing device for creating a highlight video file, comprising:
a processor; and
a non-transitory computer-readable storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
logic executed by the processor for receiving a streaming media file associated with a live broadcast of an event, said streaming media file comprising a plurality of frames of content;
logic executed by the processor for automatically analyzing each frame of the streaming media file as it is received via processor executed artificial neural network (ANN) software, said analysis comprising applying a stored index file to each individual frame using the ANN software to determine a plurality of sets of frames from the received frames that correspond to game scene and a non-game scene, said index file comprising information providing indications as to types of content that are game scenes and non-game scenes, such that the game scenes comprise content depicting game play of the event and the non-game scenes comprise content not corresponding to the game play of the event;
logic executed by the processor for analyzing the sets of frames for each game scene, said analysis comprising applying Convolutional Neural Network (CNN) image classification software to each game scene set of frames and identifying attributes of content within each game scene set of frames, said analysis further comprising computing a score via the CNN image classification software based on said attributes of the content within each set of frames;
logic executed by the processor for determining at least one highlight from said game scenes based on said analysis, said highlight determination comprising comparing said computed scores against a threshold and identifying which game scenes satisfy said threshold, the content of the threshold satisfying game scenes comprising event activity associated with deciding an outcome of the event; and
logic executed by the processor for automatically generating at least one highlight video clip based on said at least one highlight determination, each highlight video clip comprising said set of frames associated with the threshold satisfying game scenes.

2. The computing device of claim 1 , wherein said information within the index file is based on a log file associated with said event, said log file comprising information indicating all of the activity information that occurred in said event.

3. The computing device of claim 1 , wherein said information within said index file comprises labels determined from a predetermined set of machine learned attributes of content, said labels indicating whether said content is a game scene or a non-game scene, wherein said determination based on the index file further comprises applying said labels to said sets of frames.

4. The computing device of claim 1 , further comprising:
logic executed by the processor for communicating, over a network, said at least one highlight video clip to a user, said communication occurring automatically upon generation of said at least one highlight video clip.

5. The computing device of claim 1 , further comprising:
logic executed by the processor for parsing each determined plurality of set of frames and identifying data and metadata associated with each frame set, said data and metadata comprising information related to said attributes of content within each set of frames;
logic executed by the processor for translating, via processor applied vector analysis software, each set of data and metadata into individual feature vectors, each feature vector comprising information indicating a relationship of said data and metadata to said content and said set of frames; and
logic executed by the processor for storing said feature vector in a searchable database associated with said computing device.

6. The computing device of claim 1 , wherein said streaming media file further comprises an index file that controls how said plurality of frames are streamed and received by said computing device.

7. The computing device of claim 1 , further comprising:
logic executed by the processor for receiving instructions to render a stored version of the streaming media file, said instructions comprising a total playback time for said rendering of the stored version;
logic executed by the processor for determining a number of highlight and non-highlight sections in the stored media file, said determination based upon said determination of said at least one highlight from said game scenes;
logic executed by the processor for determining a playback speed of the media file that corresponds to the total playback time based on said determination of said number of highlight and non-highlight sections in the stored media file, said playback speed comprising a first playback rate associated with the non-highlight sections and a differing second playback rate for said highlight sections; and
logic executed by the processor for rendering the stored media file based on said determined playback speed.

8. The computing device of claim 1 , further comprising:
logic executed by the processor for determining a context of a highlight video clip, said context based on said event activity;
logic executed by the processor for causing communication, over a network, of said context to an advertisement platform to obtain digital advertisement content associated with said context; and
logic executed by the processor for communicating a digital content object comprising said identified digital advertisement content with said highlight video clip to a user.

9. A method comprising:
receiving, by a computing device, a streaming media file associated with a live broadcast of an event, said streaming media file comprising a plurality of frames of content;
automatically analyzing, by the computing device, each frame of the streaming media file as it is received via processor executed artificial neural network (ANN) software, said analysis comprising applying a stored index file to each individual frame using the ANN software to determine a plurality of sets of frames from the received frames that correspond to game scene and a non-game scene, said index file comprising information providing indications as to types of content that are game scenes and non-game scenes, such that the game scenes comprise content depicting game play of the event and the non-game scenes comprise content not corresponding to the game play of the event;
analyzing, by the computing device, the sets of frames for each game scene, said analysis comprising applying Convolutional Neural Network (CNN) image classification software to each game scene set of frames and identifying attributes of content within each game scene set of frames, said analysis further comprising computing a score via the CNN image classification software based on said attributes of the content within each set of frames;
determining, by the computing device, at least one highlight from said game scenes based on said analysis, said highlight determination comprising comparing said computed scores against a threshold and identifying which game scenes satisfy said threshold, the content of the threshold satisfying game scenes comprising event activity associated with deciding an outcome of the event;
automatically generating, by the computing device, at least one highlight video clip based on said at least one highlight determination, each highlight video clip comprising said set of frames associated with the threshold satisfying game scenes; and
communicating, over a network, said at least one highlight video clip to a user, said communication occurring automatically upon generation of said at le

Assignees

• Yahoo Assets Llc

Inventors

• Ni Bin
• Lieb Kirk
• Hawes Rick
• Song Yale
• Schillings Benoit
• Oughourlian Vahe
• Vallmitjana Jordi
• Nystrom Jennelle
• Ruparel Hardik
• Chen Michael
• Mathes Adam
• Balasubramanian Arunkumar
• Zhou Jian
• Edelman Matt

Classifications

• H04N21/8549

  Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} · CPC title

• G11B27/28

  by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} · CPC title

• H04N21/2387

  Stream processing in response to a playback request from an end-user, e.g. for trick-play · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04N21/4882

  for displaying messages, e.g. warnings, reminders (arrangements for providing short real-time information to substation in data switching networks H04L12/1895) · CPC title