Computer vision on broadcast video

Patent metadata

Field	Value
Publication number	US-11694358-B2
Application number	US-202017079822-A
Country	US
Kind code	B2
Filing date	Oct 26, 2020
Priority date	Jan 18, 2018
Publication date	Jul 4, 2023
Grant date	Jul 4, 2023

Abstract

Disclosed are systems and methods for improving interactions with and between computers in content searching, hosting and/or providing systems supported by or configured with devices, servers and/or platforms. The disclosed systems and methods provide an image processing framework that sub-divides computer vision techniques into three computationally efficient steps: detection, classification and matching. These steps provide an improved image processing framework that can analyze live stream data of a media file, in real-time, in order to identify and track specific digital objects depicted therein. This enables not only image processing detection results, but also the capabilities of augmenting the video stream with additional data related to the detected object.

First claim

What is claimed is:

1. A method comprising:
streaming, by a computing device over a network to a device of a user, a streaming media file comprising a plurality of frames of content;
receiving, by the computing device over the network from the user device, an indication of a request for additional content, the indication identifying a set of frames of the streaming media file;
analyzing, by the computing device, the identified set of frames, said analysis comprising identifying a set of content objects that are within the set of frames, and identifying information related to the set of frames and the set of content objects;
creating, by the computing device, a queue based on the identified information related to the set of frames and the set of content objects, the queue comprising data for each of the frames in the set of frames and data indicating where and when each content object within the set of content objects appears within each frame;
executing, by the computing device, a trained network model based on the queue, the execution being based on input comprising the information related to the set of frames and the set of content objects;
identifying, by the computing device, based on the execution, a content object from the set of content objects;
retrieving, by the computing device, augmenting information related to the content object; and
communicating, by the computing device, said augmenting information to the user device for display in conjunction with the display of the content object.

2. The method of claim 1 , wherein said communication causes the augmenting information to be displayed within a user interface (UI) portion that is displaying the streaming media file.

3. The method of claim 1 , wherein said communication causes the augmenting information to be displayed within a separate user interface (UI) portion that is displaying the streaming media file.

4. The method of claim 1 , further comprising determining, based on said analysis, attributes of the content object, wherein said identifying information is based on said attributes.

5. The method of claim 4 , wherein said attributes comprise information selected from a group consisting of: color, number, size, font, shade, angle, wrinkle and lighting value.

6. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a computing device, performs a method comprising:
streaming, by the computing device over a network to a device of a user, a streaming media file comprising a plurality of frames of content;
receiving, by the computing device over the network from the user device, an indication of a request for additional content, the indication identifying a set of frames of the streaming media file;
analyzing, by the computing device, the identified set of frames, said analysis comprising identifying a set of content objects that are within the set of frames, and identifying information related to the set of frames and the set of content objects;
creating, by the computing device, a queue based on the identified information related to the set of frames and the set of content objects, the queue comprising data for each of the frames in the set of frames and data indicating where and when each content object within the set of content objects appears within each frame;
executing, by the computing device, a trained network model based on the queue, the execution being based on input comprising the information related to the set of frames and the set of content objects;
identifying, by the computing device, based on the execution, a content object from the set of content objects;
retrieving, by the computing device, augmenting information related to the content object; and
communicating, by the computing device, said augmenting information to the user device for display in conjunction with the display of the content object.

7. The non-transitory computer-readable storage medium of claim 6 , wherein said communication causes the augmenting information to be displayed within a user interface (UI) portion that is displaying the streaming media file.

8. The non-transitory computer-readable storage medium of claim 6 , wherein said communication causes the augmenting information to be displayed within a separate user interface (UI) portion that is displaying the streaming media file.

9. The non-transitory computer-readable storage medium of claim 6 , further comprising determining, based on said analysis, attributes of the content object, wherein said identifying information is based on said attributes.

10. The non-transitory computer-readable storage medium of claim 9 , wherein said attributes comprise information selected from a group consisting of: color, number, size, font, shade, angle, wrinkle and lighting value.

11. A computing device comprising:
a processor; and
a non-transitory computer-readable storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
logic executed by the processor for streaming, over a network to a device of a user, a streaming media file comprising a plurality of frames of content;
logic executed by the processor for receiving, over the network from the user device, an indication of a request for additional content, the indication identifying a set of frames of the streaming media file;
logic executed by the processor for analyzing the identified set of frames, said analysis comprising identifying a set of content objects that are within the set of frames, and identifying information related to the set of frames and the set of content objects;
logic executed by the processor for creating a queue based on the identified information related to the set of frames and the set of content objects, the queue comprising data for each of the frames in the set of frames and data indicating where and when each content object within the set of content objects appears within each frame;
logic executed by the processor for executing a trained network model based on the queue, the execution being based on input comprising the information related to the set of frames and the set of content objects;
logic executed by the processor for identifying, based on the execution, a content object from the set of content objects;
logic executed by the processor for retrieving augmenting information related to the content object; and
logic executed by the processor for communicating said augmenting information to the user device for display in conjunction with the display of the content object.

Assignees

• Verizon Patent & Licensing Inc

Inventors

• Dassa Guy
• Song Yale
• Scholz Jeffrey
• Soares Joao Vitor Baldini
• Lee Minho

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F3/0485

  Scrolling or panning · CPC title