Systems and methods for encoding and playing back video at different frame rates using enhancement layers

Patent metadata

Field	Value
Publication number	US-10893305-B2
Application number	US-201916436429-A
Country	US
Kind code	B2
Filing date	Jun 10, 2019
Priority date	Apr 5, 2014
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

Systems and methods in accordance with embodiments of this invention provide for encoding and playing back video at different frame rates using enhancement layers. In a number of embodiments, video is encoded using temporal scalability involving the creation of a base layer at a first frame rate and an enhancement layer including additional frames enabling playback at a second higher frame rate. The second higher frame rate can also be referred to as an enhanced frame rate. In a number of embodiments, the base and enhancement layers are stored in one or more container files that contain metadata describing the enhancement layer. Based on the capabilities of a playback device, it can select the particular frame rate at which to playback encoded video.

First claim

What is claimed is:

1. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:
receive video having a plurality of access units;
encode the video into a set of layers comprising a base layer and at least one enhancement layer, wherein the encoding comprises: (i) using a temporal identifier of each access unit to determine the particular layer associated with the access unit and (ii) retaining an order of the temporal identifiers of the plurality of access units of the video; and
store the set of layers into separate container files;
wherein:
the base layer comprises a sequence of frames encoded at a first frame rate such that frames in the base layer only depend upon other frames in the base layer;
the enhancement layer comprises additional frames that, when merged with the sequence of frames in the base layer, form a sequence of frames encoded at a second frame rate;
frames in the enhancement layer are encoded such that the frames in the enhancement layer only depend upon frames in the base layer; and
the enhancement layer comprises metadata that specifies a sequential order for selecting access units from the base layer and the enhancement layer to combine the plurality of access units into a single video stream.

2. The non-transitory machine readable medium of claim 1 , wherein the temporal identifier of an access unit is stored in an initial set of bits of the access unit.

3. The non-transitory machine readable medium of claim 1 , further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

4. The non-transitory machine readable medium of claim 1 , further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

5. The non-transitory machine readable medium of claim 1 , further comprising:
receiving a request for streaming of the video;
determining playback capabilities of a device requesting the video;
streaming different sets of layers of the video based on the playback capabilities of the device, wherein (i) the base layer is streamed to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is streamed to a playback device capable of streaming the plurality of layers.

6. The non-transitory machine readable medium of claim 5 , wherein streaming the different sets of layers comprises merging the different layers using the retained order of the temporal identifiers of the plurality of access units of the video.

7. The non-transitory machine readable medium of claim 1 , further comprising:
receiving a request for downloading of the video from a playback device;
providing the encoded video to the playback device.

8. The non-transitory machine readable medium of claim 1 , wherein the frame rate of the at least one enhancement layer is an integer multiple of the base layer.

9. A source encoder, comprising:
a processor configured by a source encoder application to:
receive video having a plurality of access units;
encode the video into a set of layers comprising a base layer and at least one enhancement layer, wherein the encoding comprises: (i) using a temporal identifier of each access unit to determine the particular layer associated with the access unit and (ii) retaining an order of the temporal identifiers of the plurality of access units of the video; and
store the set of layers into separate container files;
wherein:
the base layer comprises a sequence of frames encoded at a first frame rate such that frames in the base layer only depend upon other frames in the base layer;
the enhancement layer comprises additional frames that, when merged with the sequence of frames in the base layer, form a sequence of frames encoded at a second frame rate;
frames in the enhancement layer are encoded such that the frames in the enhancement layer only depend upon frames in the base layer; and
the enhancement layer comprises metadata that specifies a sequential order for selecting access units from the base layer and the enhancement layer to combine the plurality of access units into a single video stream.

10. The source encoder of claim 9 , wherein the temporal identifier of an access unit is stored in a header of the access unit.

11. The source encoder of claim 9 , further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

12. The source encoder of claim 9 , further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

13. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:
receive video having a plurality of access units;
encode the video into a set of layers comprising a base layer and at least one enhancement layer, wherein the encoding comprises: (i) using a temporal identifier of each access unit to determine the particular layer associated with the access unit and (ii) retaining an order of the temporal identifiers of the plurality of access units of the video; and
store the set of layers into separate container files;
wherein:
the base layer comprises a sequence of frames encoded at a first frame rate such that frames in the base layer only depend upon other frames in the base layer;
the enhancement layer comprises additional frames that, when merged with the sequence of frames in the base layer, form a sequence of frames encoded at a second frame rate;
frames in the enhancement layer are encoded such that the frames in the enhancement layer only depend upon frames in the base layer, and where the frames in the enhancement layer are further constrained such that the enhancement layer does not include any frames that can be decoded without reference to other frames; and
the enhancement layer comprises metadata that specifies a sequential order for selecting access units from the base layer and the enhancement layer to combine the plurality of access units into a single video stream.

Assignees

• Divx Llc

Inventors

• Van Der Schaar Auke Sjoerd
• Funnell John

Classifications

• H04N21/234327

  by decomposing into layers, e.g. base layer and one or more enhancement layers · CPC title

• H04N21/234381 Primary

  by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping · CPC title

• H04N21/6587

  Control parameters, e.g. trick play commands, viewpoint selection · CPC title

• H04N21/85406

  involving a specific file format, e.g. MP4 format · CPC title

• H04N21/25825

  involving client display capabilities, e.g. screen resolution of a mobile phone (optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06) · CPC title