Techniques for video playback decoding surface prediction

Patent metadata

Field	Value
Publication number	US-10115377-B2
Application number	US-201515117625-A
Country	US
Kind code	B2
Filing date	Sep 24, 2015
Priority date	Sep 24, 2015
Publication date	Oct 30, 2018
Grant date	Oct 30, 2018

Abstract

Techniques are disclosed for video playback decoding surface prediction. For instance, in some embodiments, video content may be parsed for information that can be used to predict what surfaces (e.g., computer graphics shapes to be rendered, as defined by vertices specifying the location and possibly other attributes of the shape) are most likely to be accessed, for example, by a display or a graphics processing unit (GPU) in the near future. In accordance with some embodiments, these surfaces may be pre-loaded, for example, into cache memory or other desired high-bandwidth memory in advance to minimize or otherwise reduce memory access latency. In some cases, these surfaces may be entered in a list that is kept updated with each new input frame, and the surfaces in that list may be kept inside the cache (or other high-bandwidth memory) for future display or GPU access.

First claim

What is claimed is:

1. A computing device comprising:
a memory; and
a processor configured to access the memory and to
store, in a decoded picture buffer shadow, a first output surface of at least one of a plurality of video frames 1 -N of a video, wherein N represents a depth of the decoded picture buffer, and wherein the first output surface defines a first computer graphics shape to be rendered within the respective video frame 1 -N;
load a plurality of output surfaces 1 -M including the first output surface from the decoded picture buffer shadow into the memory, wherein M represents a number of output surfaces in the decoded picture buffer shadow;
determine at least one of
whether the output surface for a next decode output is already loaded into the memory and, if it is not, load that output surface into the memory, and
whether the output surfaces for reference frames of future L video frames are already in the memory and, if they are not, load those output surfaces into the memory;

send a next video frame in decode order to a media driver and/or fill a decoding result into an output surface; and
determine the output surface to be used for a decoding output, parse an incoming video frame header and calculate its picture order count, and parse headers of the incoming video frame header, thereby determining one or more reference video frames on which the one or more reference video frames depend.

2. The computing device of claim 1 , wherein the M output surfaces to be loaded into the memory include one or more of
an output surface inside a display buffer list that is to be swapped out for display,
an output surface in the decoded picture buffer with the smallest picture order count,
an output surface for the next decode output, and
an output surface utilized by a video subsequent frame for decoding reference.

3. The computing device of claim 1 , wherein the processor is configured to:
determine the output surface used by one of the video frames;
parse at least one of a sequence parameter set, a picture parameter set, and a slice header, determining picture order count for the respective video frame; and
parse all headers, determining all reference frames on which the respective video frame depends.

4. The computing device of claim 1 , wherein in loading the output surfaces from the decoded picture buffer shadow into the memory, the processor is configured to do so for one decoded picture buffer shadow array comprising N output surfaces and sorted by picture order count.

5. The computing device of claim 1 , wherein the processor is further configured to:
add a new output surface into the decoded picture buffer and reorder the decoded picture buffer shadow against a picture order count.

6. The computing device of claim 5 , wherein the processor is further configured to:
determine whether the first M output surfaces in the decoded picture buffer shadow have changed and, if so, load a newcomer output surface into the memory and evict a dispensable output surface against the picture order count;
determine whether the output surface for a next decode output is already loaded into the memory and, if it is not, load that output surface into the memory; and
determine whether one or more of the output surfaces for reference video frames of future L video frames are already in the memory and, if they are not, load those output surfaces into the memory.

7. The computing device of claim 1 , wherein at least one of:
the memory comprises cache memory; and
the processor comprises a graphics processing unit.

8. The computing device of claim 1 , wherein the memory comprises media middleware configured to generate the decoded picture buffer shadow.

9. A method of decoding video content, the method comprising:
storing, in a decoded picture buffer shadow, a first output surface of at least one of a plurality of video frames 1 -N of a video, wherein N represents a depth of the decoded picture buffer, and wherein the first output surface defines a first computer graphics shape to be rendered within the respective video frame 1 -N;
loading a plurality of output surfaces 1 -M including the first output surface from the decoded picture buffer shadow into the memory, wherein M represents a number of output surfaces in the decoded picture buffer shadow;
determine at least one of
whether the output surface for a next decode output is already loaded into the memory and, if it is not, load that output surface into the memory, and
whether the output surfaces for reference frames of future L video frames are already in the memory and, if they are not, load those output surfaces into the memory;

send a next video frame in decode order to a media driver and/or fill a decoding result into an output surface; and
determine the output surface to be used for a decoding output, parse an incoming video frame header and calculate its picture order count, and parse headers of the incoming video frame header, thereby determining one or more reference video frames on which the one or more reference video frames depend.

10. The method of claim 9 , wherein the M output surfaces to be loaded into the memory include one or more of
an output surface inside a display buffer list that is to be swapped out for display,
an output surface in the decoded picture buffer with the smallest picture order count,
an output surface for the next decode output, and
an output surface utilized by a subsequent video frame for decoding reference.

11. The method of claim 9 , further comprising:
determining the output surface used by one of the video frames;
parsing at least one of a sequence parameter set, a picture parameter set, and a slice header, determining picture order count for the respective video frame; and
parsing all headers, determining all reference frames on which the respective video frame depends.

12. The method of claim 9 , wherein loading the output surfaces from the decoded picture buffer shadow into the memory is performed for one decoded picture buffer shadow array comprising N output surfaces and sorted by picture order count.

13. The method of claim 9 further comprising:
adding a new output surface into the decoded picture buffer and reordering the decoded picture buffer shadow against a picture order count.

14. The method of claim 13 further comprising:
determining whether the first M output surfaces in the decoded picture buffer shadow have changed and, if so, loading a newcomer output surface into the memory and evicting a dispensable output surface against the picture order count;
determining whether the output surface for a next decode output is already loaded into the memory and, if it is not, loading that output surface into the memory; and
determining whether one or more of the output surfaces for reference video frames of future L video frames are already in the memory and, if they are not, loading those output surfaces into the memory.

15. The method of claim 9 , wherein the decoded picture buffer shadow is at least one of:
established during initialization and refreshed after each new input; and
generated via media middleware.

16. A non-transitory computer program product encoded with instructions that, when executed by one or more processors, causes a process to be carried out, the process comprising:
store, in a decoded picture buffer shadow, a first output surface of at least one of a plurality of video frames 1 -N of a video, wherein N represents a depth of the decoded picture buffer, and wherein the first output surface defines a first computer gra

Assignees

• Intel Corp

Inventors

• Luo Ning
• Wang Changliang
• Lee Penne Y

Classifications

• H04N19/423 Primary

  characterised by memory arrangements (H04N19/433 takes precedence) · CPC title

• G09G2360/121

  using a cache memory · CPC title

• H04N19/44

  Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder · CPC title

• G09G5/395 Primary

  Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen (G09G5/399 takes precedence) · CPC title

• G09G2360/10

  Display system comprising arrangements, such as a coprocessor, specific for motion video images · CPC title