Computerized system and method for fine-grained event detection and content hosting therefrom

Patent metadata

Field	Value
Publication number	US-12579808-B2
Application number	US-202418769976-A
Country	US
Kind code	B2
Filing date	Jul 11, 2024
Priority date	Mar 24, 2021
Publication date	Mar 17, 2026
Grant date	Mar 17, 2026

Abstract

The disclosed systems and methods provide a novel framework that provides mechanisms for performing cost-effective, accurate and scalable detection and recognition of fine-grained events. The framework functions by training high precision and high recall object/optical character recognition (OCR) models and aligning video frames to text commentaries of the videos (e.g., licensed play-by-play). The disclosed framework operates as a single algorithm that performs multimodal alignments between events/actions within videos and their prescribed text. Thus, the disclosed framework is able to scale to fine-grained action categories across different venues by delving into the key frames and key aspects of a video to identify particular actions performed by particular actors, thereby providing the novelty of fine-granted action detection and recognition.

First claim

What is claimed is:

1 . A method comprising:
identifying a video, the video comprising content corresponding to an event; analyzing the video, and determining attributes of the video, the attributes corresponding to characteristics of playback of the video when rendered; performing a batch prediction, based on the determined attributes of the video, the batch prediction corresponding to a position within frames of the video related to a clock; determining, based on further analysis of the video, a set of frames corresponding to an action represented within the content of the video; analyzing the set of frames based on the batch prediction, and determining a time period associated with the action; and generating a video segment as a highlight of the video, the video segment comprising the set of frames corresponding to the determined time period.

2 . The method of claim 1 , further comprising:
analyzing, via a multimedia stream analyzer, the video by probing the video to identify the characteristics of playback, the characteristics of playback comprising at least one of resolution, frame rate, image sequences, frame height and width, length of the video and publish date.

3 . The method of claim 1 , further comprising:
decoding the frames of the video via a single shot detector (SSD) model; and determining the set of frames based further on the decoding via the SSD model.

4 . The method of claim 1 , further comprising:
identifying play-by-play text related to the video; and performing the further analysis of the video based on the play-by-play text.

5 . The method of claim 1 , further comprising:
analyzing the frames of the video, and determining bounding boxes within each frame corresponding to the clock; clustering the set of frames based on the bounding boxes; and generating the video segment based on the clustering.

6 . The method of claim 5 , further comprising:
performing text recognition within the bounding boxes within each frame; and performing the clustering based on a result of the text recognition for the set of frames.

7 . The method of claim 1 , further comprising:
receiving a request for content related to an item within the video; and generating the video segment based on the item identified by the request, wherein the item corresponds to at least one of a type of action, a particular player, a particular time period, and a particular team.

8 . The method of claim 1 , further comprising:
requesting, over a network, third party digital content based at least on information related to the video segment; receiving, over the network, the third party digital content; and communicating, over the network, the third party digital content for display along with the video segments.

9 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a device, performs a method comprising:
identifying a video, the video comprising content corresponding to an event; analyzing the video, and determining attributes of the video, the attributes corresponding to characteristics of playback of the video when rendered; performing a batch prediction, based on the determined attributes of the video, the batch prediction corresponding to a position within frames of the video related to a clock; determining, based on further analysis of the video, a set of frames corresponding to an action represented within the content of the video; analyzing the set of frames based on the batch prediction, and determining a time period associated with the action; and generating a video segment as a highlight of the video, the video segment comprising the set of frames corresponding to the determined time period.

10 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
analyzing, via a multimedia stream analyzer, the video by probing the video to identify the characteristics of playback, the characteristics of playback comprising at least one of resolution, frame rate, image sequences, frame height and width, length of the video and publish date.

11 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
decoding the frames of the video via a single shot detector (SSD) model; and determining the set of frames based further on the decoding via the SSD model.

12 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
identifying play-by-play text related to the video; and performing the further analysis of the video based on the play-by-play text.

13 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
analyzing the frames of the video, and determining bounding boxes within each frame corresponding to the clock; clustering the set of frames based on the bounding boxes; and generating the video segment based on the clustering.

14 . The non-transitory computer-readable storage medium of claim 13 , further comprising:
performing text recognition within the bounding boxes within each frame; and performing the clustering based on a result of the text recognition for the set of frames.

15 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
receiving a request for content related to an item within the video; and generating the video segment based on the item identified by the request, wherein the item corresponds to at least one of a type of action, a particular player, a particular time period, and a particular team.

16 . A device comprising:
a processor configured to:
identify a video, the video comprising content corresponding to an event;
analyze the video, and determine attributes of the video, the attributes corresponding to characteristics of playback of the video when rendered;
perform a batch prediction, based on the determined attributes of the video, the batch prediction corresponding to a position within frames of the video related to a clock;
determine, based on further analysis of the video, a set of frames corresponding to an action represented within the content of the video;
analyze the set of frames based on the batch prediction, and determine a time period associated with the action; and
generate a video segment as a highlight of the video, the video segment comprising the set of frames corresponding to the determined time period.

17 . The device of claim 16 , wherein the processor is configured to:
analyze, via a multimedia stream analyzer, the video by probing the video to identify the characteristics of playback, the characteristics of playback comprising at least one of resolution, frame rate, image sequences, frame height and width, length of the video and publish date.

18 . The device of claim 16 , wherein the processor is configured to:
decode the frames of the video via a single shot detector (SSD) model; and determine the set of frames based further on the decoding via the SSD model.

19 . The device of claim 16 , wherein the processor is configured to:
identify play-by-play text related to the video; and perform the further analysis of the video based on the play-by-play text.

20 . The device of claim 16 , wherein the processor is configured to:
analyze the frames of the video, and determine bounding boxes within each frame corresponding to the clock; cluster the set of frames based on the bounding boxes; and generate the video segment

Assignees

• Yahoo Assets Llc

Inventors

• Biswas Topojoy
• Shah Avijit
• Shah Deven Santosh

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06V30/10

  Character recognition · CPC title

• G06V20/44

  Event detection · CPC title

• G06V20/63

  Scene text, e.g. street names · CPC title