Error detection, protection and recovery for video decoding

Patent metadata

Field	Value
Publication number	US-9924184-B2
Application number	US-21703708-A
Country	US
Kind code	B2
Filing date	Jun 30, 2008
Priority date	Jun 30, 2008
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

Error detecting and protection innovations for video decoders are described. For example, in a multithreaded video decoder, a picture extent discovery (PED) task detects an error in a video bitstream which corrupts a picture. The PED task then determines any PED sub-stage which have been completed for the picture, and based on this determination, performs error-handing PED operations. In another example, an entropy decoding (ED) task checks validity on a macroblock-by-macroblock basis using a redundant buffer to avoid overflows. Additionally, error recovery innovations are described which facilitate playback of a video bit stream at an arbitrary position. For example, a video decoder chooses a picture in the bit stream after the arbitrary position at which to begin decoding based on a determination of acceptable recovery time and/or acceptable picture quality.

First claim

We claim:

1. A method for handling an error in a video bit stream in a multi-threaded video decoding system, the method comprising:
before entering a picture extent discovery (PED) task with PED sub-stages, parsing the bit stream and verifying that only network abstraction layer units (NALUs) which have proper start codes and which are within a predetermined maximum size are accepted as valid data;
during the PED task with PED sub-stages, detecting in the bit stream an error that corrupts a picture, wherein the PED sub-stages include a sub-stage for initializing structures for decoding the picture and a sub-stage for entering the initialized but not yet decoded picture in a decoded picture buffer (DPB) to facilitate management of the DPB, during decoding of plural pictures in flight at various stages of multi-threaded decoding, that simulates management of the DPB as in single-threaded decoding;
determining which of the PED sub-stages, if any, have been completed for the corrupted picture; and
based on the PED sub-stages, if any, which have been completed, selecting and performing one or more PED operations to handle the error.

2. The method of claim 1 , wherein the PED sub-stages further include a sub-stage for tracking dependencies for entry of tasks into a task scheduling data structure.

3. The method of claim 2 , wherein:
tasks for the picture are determined to have entered the task scheduling data structure and the picture is determined to have entered the DPB; and
the one or more PED operations to handle the error comprise modifying the DPB to have a correct state and to show the picture as corrupt.

4. The method of claim 2 , wherein:
tasks for the picture are determined to have entered the task scheduling data structure but the picture is determined not to have entered the DPB; and
the one or more PED operations to handle the error comprise:
adjusting settings for the picture such that subsequent decoding tasks are not performed as the picture goes through a decoding process; and
adding picture delete commands to a command queue for the picture.

5. The method of claim 2 , wherein:
structures for the picture are determined to have not yet been initialized for decoding; and
the one or more PED operations to handle the error comprise:
resetting flags for the picture; and
checking a picture command queue for commands to execute.

6. The method of claim 2 , wherein:
structures for the picture are determined to have been initialized but tasks for the picture are determined not to have entered the task scheduling data structure; and
the one or more PED operations to handle the error comprise recycling resources used by the picture and resetting flags for the picture.

7. The method of claim 1 , further comprising, during the PED task:
verifying syntax of sequence parameter set, picture parameter set, and supplemental enhancement information NALUs; and
discarding NALUs which fail syntax verification.

8. The method of claim 1 , further comprising, during the PED task:
receiving indications from transport layer or network layer tasks that parameters in NALUs are corrupted; and
discarding NALUs which are identified as having corrupted parameters.

9. The method of claim 1 , further comprising, during an entropy decoding task for at least part of a subsequent picture, checking validity of entropy decoding on a macroblock-by-macroblock basis for each of plural macroblocks after each macroblock is decoded.

10. The method of claim 1 , further comprising, for at least part of a subsequent picture, performing an entropy decoding task utilizing a redundant buffer comprising an overflow area and, if a decoded macroblock enters the overflow area, terminating the entropy decoding task.

11. The method of claim 1 , further comprising, for at least part of a subsequent picture, performing syntax verifications during an entropy decoding task and, when syntax verifications indicate a corrupted syntax element, terminating the entropy decoding task.

12. The method of claim 1 , further comprising, during an entropy decoding task for at least part of a subsequent picture, verifying a number of actually decoded macroblocks against a predicted number of macroblocks determined during the PED task for the subsequent picture.

13. The method of claim 1 , further comprising, during decoding of a subsequent picture:
identifying a macroblock as corrupted during motion compensation or intra prediction;
marking the identified macroblock as corrupted; and
disabling deblocking across a boundary of the macroblock.

14. A multi-threaded video decoding system comprising one or more processors, memory and software for causing the system to perform a method of handling an error in a video bit stream, the software comprising:
code for, before entering a picture extent discovery (PED) task with PED sub-stages, parsing the bit stream and verifying that only network abstraction layer units (NALUs) which have proper start codes and which are within a predetermined maximum size are accepted as valid data;
code for, during the PED task with PED sub-stages, detecting in the bit stream an error that corrupts a picture, wherein the PED sub-stages include a sub-stage for initializing structures for decoding the picture and a sub-stage for entering the initialized but not yet decoded picture in a decoded picture buffer (DPB) to facilitate management of the DPB, during decoding of plural pictures in flight at various stages of multi-threaded decoding, that simulates management of the DPB as in single-threaded decoding;
code for determining which of the PED sub-stages, if any, have been completed for the corrupted picture; and
code for, based on the PED sub-stages, if any, which have been completed, selecting and performing one or more PED operations to handle the error.

15. The system of claim 14 , wherein the PED sub-stages further include a sub-stage for tracking dependencies for entry of tasks into a task scheduling data structure.

16. The system of claim 15 , wherein, when tasks for the picture are determined to have entered the task scheduling data structure and the picture is determined to have entered the DPB, the one or more PED operations to handle the error comprise modifying the DPB to have a correct state and to show the picture as corrupt.

17. The system of claim 15 , wherein, when tasks for the picture are determined to have entered the task scheduling data structure but the picture is determined not to have entered the DPB, the one or more PED operations to handle the error comprise:
adjusting settings for the picture such that subsequent decoding tasks are not performed as the picture goes through a decoding process; and
adding picture delete commands to a command queue for the picture.

18. The system of claim 15 , wherein, when structures for the picture are determined to have not yet been initialized for decoding, the one or more PED operations to handle the error comprise:
resetting flags for the picture; and
checking a picture command queue for commands to execute.

19. The system of claim 15 , wherein, when structures for the picture are determined to have been initialized but tasks for the picture are determined not to have entered the task scheduling data structure, the one or more PED operations to handle the error comprise recycling resources used by the picture and resetting flags for the picture.

20. The system of claim 14 , wherein the PED task further includes:
verifying syntax of sequence parameter

Assignees

• Wu Yongjun
• Christoffersen Eric S
• Thumpudi Naveen
• Microsoft Technology Licensing Llc

Inventors

• Wu Yongjun
• Christoffersen Eric S
• Thumpudi Naveen

Classifications

• H04N19/89

  involving methods or arrangements for detection of transmission errors at the decoder · CPC title

• H04N19/61

  in combination with predictive coding · CPC title

• H04N19/44 Primary

  Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder · CPC title