Computer vision on broadcast video

Patent metadata

Field	Value
Publication number	US-10818033-B2
Application number	US-201815907586-A
Country	US
Kind code	B2
Filing date	Feb 28, 2018
Priority date	Jan 18, 2018
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

Disclosed are systems and methods for improving interactions with and between computers in content searching, hosting and/or providing systems supported by or configured with devices, servers and/or platforms. The disclosed systems and methods provide an image processing framework that sub-divides computer vision techniques into three computationally efficient steps: detection, classification and matching. These steps provide an improved image processing framework that can analyze live stream data of a media file, in real-time, in order to identify and track specific digital objects depicted therein. This enables not only image processing detection results, but also the capabilities of augmenting the video stream with additional data related to the detected object.

First claim

What is claimed is:

1. A method comprising:
receiving, at a computing device, a streaming media file comprising a plurality of frames of content to be displayed within a first portion of a user interface (UI) displayed on a display of the computing device;
analyzing, via the computing device, the content of the received frames of the streaming media file, and based on said analysis, determining a first frame depicting a first element, said determination comprising identifying a display location of the first element within the first frame;
identifying, via the computing device, based on said determination, an image portion of the first frame, said identified image portion comprising a portion of the content within the first frame that corresponds to the display location of the first element within the first frame;
executing, via the computing device, image classifier software on the image portion, said execution comprising analyzing attributes of the image portion and performing image classification of the image portion based on a synthetic training image dataset, and based on said analysis and classification, determining information indicating an identity and type of the first element;
retrieving, via the computing device, augmenting information to the streaming media file that corresponds to the first element based on said determined information of the first element; and
displaying, via the computing device, said augmenting information within a separate second portion of the UI while said first frame is displayed within said first portion of the UI, the second portion being distinct and separate from the first portion.

2. The method of claim 1 , wherein said identification of the image portion comprises:
creating an image patch, said creation comprising cropping the first frame such that each image patch comprises only the content depicted within a portion of said first frame.

3. The method of claim 2 , further comprising:
applying a bounding box to the first frame in accordance with said display location of the first element, wherein said cropping is based upon said application of the bounding box.

4. The method of claim 2 , wherein said creation of said image patch is executed by the computing device executing feature extraction software on the first frame.

5. The method of claim 1 , further comprising:
analyzing the received streaming media file, and based on said analysis, identifying a frame set that corresponds to a particular scene, said determination of said first element corresponds to identification of a display location within each frame in said frame set.

6. The method of claim 5 , wherein said identification of the frame set is based on the computing device executing Fast Forward MPEG (FFMPEG) software on said streaming media file.

7. The method of claim 1 , further comprising:
determining, based on said analysis of said content of the received frames, a number of sequential frames in which the first element appears;
determining whether said number of sequential frames satisfies an occurrence threshold; and
discarding a segment of the media file when said threshold determination is not satisfactory prior to identification of an associated image portion.

8. The method of claim 1 , wherein said image classifier software comprises a trained network model, wherein training of the model comprises:
generating a synthetic dataset of images depicting specific content elements;
identifying a real image dataset depicting said specific content elements;
executing data manipulation software on both image sets, said executing causing attributes of each image in the synthetic dataset to be modified to correspond to attributes of the real image dataset; and
generating the synthetic training image dataset based on said data manipulation software execution, said training image dataset forming a definition within said image classifier software.

9. The method of claim 8 , wherein said trained network model is a Convolutional Neural Network (CNN).

10. The method of claim 1 , wherein said first element corresponds to a depiction of a jersey number within said first frame, wherein said identity corresponds to a specific jersey number, wherein said type corresponds to attributes of said jersey number and a jersey upon which it is displayed.

11. The method of claim 10 , wherein said attributes of the jersey and jersey number comprise characteristics selected from a group consisting of: fonts, shapes, colors, shades, lighting values, display angles and wrinkles.

12. The method of claim 1 , wherein said streaming media file is associated with a live broadcast of an event.

13. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a computing device, performs a method comprising:
receiving, at the computing device, a streaming media file comprising a plurality of frames of content to be displayed within a first portion of a user interface (UI) displayed on a display of the computing device;
analyzing, via the computing device, the content of the received frames of the streaming media file, and based on said analysis, determining a first frame depicting a first element, said determination comprising identifying a display location of the first element within the first frame;
identifying, via the computing device, based on said determination, an image portion of the first frame, said identified image portion comprising a portion of the content within the first frame that corresponds to the display location of the first element within the first frame;
executing, via the computing device, image classifier software on the image portion, said execution comprising analyzing attributes of the image portion and performing image classification of the image portion based on a synthetic training image dataset, and based on said analysis and classification, determining information indicating an identity and type of the first element;
retrieving, via the computing device, augmenting information to the streaming media file that corresponds to the first element based on said determined information of the first element; and
displaying, via the computing device, said augmenting information within a separate second portion of the UI while said first frame is displayed within said first portion of the UI, the second portion being distinct and separate from the first portion.

14. The non-transitory computer-readable storage medium of claim 13 , wherein said identification of the image portion comprises:
creating an image patch, said creation comprising cropping the first frame such that each image patch comprises only the content depicted within a portion of said first frame.

15. The non-transitory computer-readable storage medium of claim 14 , further comprising:
applying a bounding box to the first frame in accordance with said display location of the first element, wherein said cropping is based upon said application of the bounding box.

16. The non-transitory computer-readable storage medium of claim 13 , further comprising:
analyzing the received streaming media file, and based on said analysis, identifying a frame set that corresponds to a particular scene, said determination of said first element corresponds to identification of a display location within each frame in said frame set.

17. The non-transitory computer-readable storage medium of claim 13 , further comprising:
determining, based on said analysis of said content of the received frames, a number of sequential frame

Assignees

• Oath Inc

Inventors

• Dassa Guy
• Song Yale
• Scholz Jeffrey
• Soares Joao Vitor Baldini
• Lee Minho

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title