Systems and methods for encoding and playing back video at different frame rates using enhancement layers

Patent metadata

Field	Value
Publication number	US-9866878-B2
Application number	US-201514679469-A
Country	US
Kind code	B2
Filing date	Apr 6, 2015
Priority date	Apr 5, 2014
Publication date	Jan 9, 2018
Grant date	Jan 9, 2018

Abstract

Systems and methods in accordance with embodiments of this invention provide for encoding and playing back video at different frame rates using enhancement layers. In a number of embodiments, video is encoded using temporal scalability involving the creation of a base layer at a first frame rate and an enhancement layer including additional frames enabling playback at a second higher frame rate. The second higher frame rate can also be referred to as an enhanced frame rate. In a number of embodiments, the base and enhancement layers are stored in one or more container files that contain metadata describing the enhancement layer. Based on the capabilities of a playback device, it can select the particular frame rate at which to playback encoded video.

First claim

What is claimed is:

1. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:
obtain video comprising a merged plurality of access units with temporal scalability, each access unit in the merged plurality of access units associated with a particular layer of a plurality of temporal layers based on a temporal identifier of the access unit that identifies the particular layer, wherein the temporal layers include a base layer and one or more enhancement layers;
obtain an ordered sequence of access units ordered based on a decoding order for decoding the video and generating metadata specifying an ordered sequence of temporal identifiers ordered based on, for each access unit in the ordered sequence of access units, identifying and retaining the temporal identifier associated with the access unit;
separate, based on the temporal identifier of each access unit in the ordered sequence of access units, the ordered sequence of access units into (1) a base layer of access units sequenced in decode order to provide a base frame rate and (2) at least one enhancement layer with additional access units ordered in decode order to provide an enhanced frame rate, wherein at least some of the access units in the enhancement layer are encoded based upon dependencies to access units in the base layer, and wherein access units in the base layer are encoded without dependencies upon access units in the enhancement layer; and
storing the base layer in a first container file; and
storing the enhancement layer, including the generated metadata specifying the ordered sequence of temporal identifiers, in a second container file separate from the first container file, wherein the metadata is used to recombine in decode order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

2. The non-transitory machine readable medium of claim 1 , wherein the temporal identifier of an access unit is stored in an initial set of bits of the access unit.

3. The non-transitory machine readable medium of claim 1 , further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

4. The non-transitory machine readable medium of claim 1 , further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

5. The non-transitory machine readable medium of claim 1 , further comprising:
receiving a request for streaming of the video;
determining playback capabilities of a device requesting the video;
streaming different sets of layers of the video based on the playback capabilities of the device, wherein (i) the base layer is streamed to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is streamed to a playback device capable of streaming the plurality of layers.

6. The non-transitory machine readable medium of claim 5 , wherein streaming the different sets of layers comprises merging the different layers using the metadata that specifies the sequence for combining access units from the base layer with access units from the enhancement layer.

7. The non-transitory machine readable medium of claim 1 , further comprising:
receiving a request for downloading of the video from a playback device;
providing the encoded video to the playback device.

8. The non-transitory machine readable medium of claim 1 , wherein the frame rate of the at least one enhancement layer is an integer multiple of the base layer.

9. The non-transitory machine readable medium of claim 1 , further comprising storing each layer as a separate stream.

10. The non-transitory machine readable medium of claim 1 , further comprising storing each layer in a separate container file.

11. A source encoder, comprising:
a processor configured by a source encoder application to:
obtain video comprising a merged plurality of access units with temporal scalability, each access unit in the merged plurality of access units associated with a particular layer of a plurality of temporal layers based on a temporal identifier of the access unit that identifies the particular layer, wherein the temporal layers include a base layer and one or more enhancement layer;
obtain an ordered sequence of access units ordered based on a decoding order for decoding the video and generating metadata specifying an ordered sequence of temporal identifiers ordered based on, for each access unit in the ordered sequence of access units, identifying and retaining the temporal identifier associated with the access unit;
separate, based on the temporal identifier of each access unit in the ordered sequence of access units, the ordered sequence of access units into (1) a base layer of access units sequenced in decode order to provide a base frame rate and (2) at least one enhancement layer with additional access units ordered in decode order to provide an enhanced frame rate, wherein at least some of the access units in the enhancement layer are encoded based upon dependencies to access units in the base layer, and wherein access units in the base layer are encoded without dependencies upon access units in the enhancement layer;
storing the base layer in a first container file; and
storing the enhancement layer, including the generated metadata specifying the ordered sequence of temporal identifiers, in a second container file separate from the first container file, wherein the metadata is used to recombine in decoding order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

12. The source encoder of claim 11 , wherein the temporal identifier of an access unit is stored in a header of the access unit.

13. The source encoder of claim 11 , further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

14. The source encoder of claim 11 , further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

15. A playback device configured to playback video comprising:
a processor configured to communicate with a memory, where the memory contains a client application;
wherein the client application configures the processor to:
obtain a first container file comprising a base layer of access units ordered in decode order for playback at a base frame rate;
obtain, for a device capable of playing back an enhancement layer, a second container file comprising (1) at least one enhancement layer with additional access units ordered in decode order for playback at an enhanced frame rate and (2) metadata specifying an ordered sequence of temporal identifiers for combining, in decode order, access units from the base layer with access units from the at least one enhancement layer for playback at the enhanced frame rate, wherein each access unit includes a temporal identifier identifying a particular layer associated with the access unit;
combine the base layer of access units with the at least one enhancement layer based on the metadata obtained from the second container file to obtain a combined sequence of access units ordered in a decoding order at the enhanced frame rate;
wherein at least some of the access units in the at least enhancement layer stored in the second container file are decoded based upon dependencies to acc

Assignees

• Sonic Ip Inc

Inventors

• Van Der Schaar Auke Sjoerd
• Funnell John

Classifications

• H04N21/234327

  by decomposing into layers, e.g. base layer and one or more enhancement layers · CPC title

• H04N21/6587

  Control parameters, e.g. trick play commands, viewpoint selection · CPC title

• H04N21/234381 Primary

  by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping · CPC title

• H04N21/85406

  involving a specific file format, e.g. MP4 format · CPC title

• H04N21/25825

  involving client display capabilities, e.g. screen resolution of a mobile phone (optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06) · CPC title