System and method for controlling data stream modifications

Patent metadata

Field	Value
Publication number	US-11375249-B2
Application number	US-201716322850-A
Country	US
Kind code	B2
Filing date	Aug 3, 2017
Priority date	Aug 3, 2016
Publication date	Jun 28, 2022
Grant date	Jun 28, 2022

Abstract

In some embodiments, there is provided a system for remotely processing data streams including: at least one transmitter configured for transmission of at least two data streams, the at least two data streams including at least a lower quality preview stream and a higher quality content stream. A plurality of editor computing devices is configured for receiving at least the lower quality preview stream and facilitating processing and editing on the lower quality preview stream, generating a set of machine-readable instructions representative of the processing and editing. A plurality of routing computing devices located from the at least one transmitter is configured for receiving at least the higher quality content stream and the set of machine-readable instructions, processing the higher quality content stream by encoding the higher quality content stream based on the set of machine-readable instructions to generate an output content stream.

First claim

The invention claimed is:

1. A system for controlling data stream modifications, the system comprising:
a router configured to receive a preview data stream, and to generate, based on the preview data stream, one or more sets of processing instructions for modifying a pre-processed data stream;
a data stream encoder configured to receive the pre-processed data stream and to apply one or more data modifications based at least on executing the one or more sets of processing instructions in processing the pre-processed data stream, the data stream encoder generating an output processed stream including the data modifications;
wherein the preview data stream and the pre-processed data stream are both derived from a source video data stream;
wherein receipt of the corresponding frames of the preview data stream and the corresponding frames of the pre-processed data stream is temporally shifted relative to one another at least in part due to one or more encoding differences between the preview data stream and the pre-processed data stream; and
wherein the receipt of corresponding frames of the preview data stream by the router occurs ahead of the receipt of corresponding frames of the pre-processed data stream by the data stream encoder.

2. The system of claim 1 , wherein the pre-processed data stream is a production quality data stream generated from the source video data stream.

3. The system of claim 1 , comprising:
at least one receiver configured to receive the preview data stream and the pre-processed data stream as separate data streams.

4. The system of claim 1 , comprising:
at least one receiver configured to receive the source video data stream; and
a first encoder configured to generate the preview data stream from the source video data stream; and
a second encoder configured to generate the pre-processed data stream from the source video data stream;
wherein the preview data stream is generated at a lower bitrate than the pre-processed data stream.

5. The system of claim 4 , wherein the second encoder delays transmission of the pre-processed data stream to the data stream encoder to cause temporal shifting of the receipt of the corresponding frames of the pre-processed data stream relative to the corresponding frames of the preview data stream.

6. The system of claim 4 , wherein operating characteristics of at least one of the first encoder and the second encoder are controllable to modify a time delay between the receipt of the corresponding frames of the preview data stream and the corresponding frames of the pre-processed data stream.

7. The system of claim 1 , wherein the router and the data stream encoder are configured for operation in a master-slave configuration, whereby the one or more sets of processing instructions represent uni-directional processing steps for execution by the data stream encoder;
wherein differences in encoding between the preview data stream and the pre-processed data stream result in a minimum time delay between the receipt of the corresponding frames of the preview data stream and the corresponding frames of the pre-processed data stream; and
wherein the router is configured for identifying data modifications and generating the one or more sets of processing instructions representing the identified data modifications, which when executed by the data stream encoder, transform the pre-processed data stream into the output processed stream.

8. The system of claim 7 , wherein the router is configured to generate the one or more sets of processing instructions within a temporal constraint defined at least by the temporal shift between the receipt the corresponding frames of the preview data stream and the corresponding frames of the pre-processed data stream.

9. The system of claim 8 , wherein each set of the one or more sets of processing instructions for modifying the pre-processed data stream includes at least a data structure having frame identifiers of frames of the pre-processed data stream selected for modification, and image or audio manipulation instructions associated with the selected frames of the pre-processed data stream.

10. The system of claim 1 , wherein the router is further configured to transmit at least one of the preview data stream and the one or more sets of processing instructions for rendering on a preview visualization display screen.

11. The system of claim 8 , comprising:
a orchestration coordination router configured for coordinating processing of the pre-processed data stream such that the generation of the output processed stream including the data modifications occurs within the temporal constraint.

12. The system of claim 11 , wherein the orchestration coordination router is configured for selecting, from a plurality of cloud computing resources, a subset of the plurality of cloud computing resources for implementing or for use by at least one of the router and the data stream encoder.

13. The system of claim 12 , wherein the orchestration coordination router is further configured to control one or more networking connections between the cloud computing resources of the plurality of cloud computing resources, the orchestration coordination router controlling the one or more networking connections to modify an overall time required for generating one or more sets of processing instructions, the overall time falling within the temporal constraint.

14. The system of claim 13 , wherein each cloud computing resource of the plurality of cloud computing resources is associated with corresponding geospatial characteristics, and wherein the orchestration coordination router, selects the subset of the plurality of cloud computing resources based at least in part on the geospatial characteristics of each cloud computing resource.

15. The system of claim 14 , wherein the orchestration coordination router is configured to allocate specific cloud computing resources when required for processing, and configured to release the allocated cloud computing resources when the processing is complete.

16. The system of claim 15 , wherein the orchestration coordination router is configured to allocate and subsequently release a rolling set of cloud computing resources selected from time-zone regions having off-peak usage.

17. The system of claim 14 , wherein the geospatial characteristics of each cloud computing resource includes at least one of a geographical location, a time zone, and a networking latency for communications to and from the corresponding cloud computing resource.

18. The system of claim 17 , wherein the orchestration coordination router, upon a determination that the processing is likely to exceed the temporal constraint given a present set of conditions, is configured to request prioritized networking resources.

19. The system of claim 11 , wherein the orchestration coordination router is configured to control a duration of the temporal shift between the preview data stream and the pre-processed data stream.

20. The system of claim 12 , wherein the generation of the one or more sets of processing instructions is conducted across a chain of cloud computing resources formed by the subset of the plurality of cloud computing resources, and wherein an expected end to end latency is periodically determined by aggregating a network latency from a source transmitter, a processing time required at each of the cloud computing resources, one or more network latencies between each of the cloud computing resources, and a network latency to the data stream encoder;

Assignees

• Dejero Labs Inc

Inventors

• Sze David
• Oberholzer Jonathon
• Schneider Todd
• Frusina Bogdan

Classifications

• H04L65/765

  intermediate · CPC title

• H04N21/8549

  Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} · CPC title

• H04N21/234

  Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs · CPC title

• H04N21/854

  Content authoring · CPC title

• H04N21/2343 Primary

  involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} · CPC title