Use of frame caching to improve packet loss recovery

Patent metadata

Field	Value
Publication number	US-9313501-B2
Application number	US-201213614453-A
Country	US
Kind code	B2
Filing date	Sep 13, 2012
Priority date	Dec 30, 2004
Publication date	Apr 12, 2016
Grant date	Apr 12, 2016

Abstract

Various new and non-obvious apparatus and methods for using frame caching to improve packet loss recovery are disclosed. One of the disclosed embodiments is a method for using periodical and synchronized frame caching within an encoder and its corresponding decoder. When the decoder discovers packet loss, it informs the encoder which then generates a frame based on one of the shared frames stored at both the encoder and the decoder. When the decoder receives this generated frame it can decode it using its locally cached frame.

First claim

We claim:

1. In a computing system that implements a video decoder, a method of decoding plural frames of a video sequence, the method comprising:
decoding a first frame of the plural frames using encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;
storing, for use as a first reference frame, the first frame in a first portion of a decoder-side frame cache that corresponds to a first portion of an encoder-side frame cache at a video encoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;
decoding a second frame of the plural frames using encoded data for the second frame, the second frame following the first frame in the video sequence;
storing, for use as a second reference frame, the second frame in a second portion of the decoder-side frame cache different than the first portion of the decoder-side frame cache;
detecting loss of encoded data for one of the plural frames;
in response to the detecting loss of encoded data, transmitting a loss detection signal in order to cause the video encoder to generate a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder, the new frame including different data than the one of the plural frames for which loss of encoded data is detected; and
decoding the new frame using encoded data for the new frame and using the first reference frame stored in the first portion of the decoder-side frame cache, the encoded data for the new frame having been generated by the video encoder, in response to the loss detection signal, during encoding of the new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder.

2. The method of claim 1 wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in a second portion of the encoder-side frame cache at the video encoder.

3. The method of claim 1 wherein the decoding the new frame includes:
arithmetic decoding encoded transform coefficients and encoded side information in the encoded data;
inverse quantizing the transform coefficients to produce dequantized transform coefficients;
inverse frequency transforming the dequantized transform coefficients to produce residual values;
performing motion compensation to produce motion-compensated prediction values using motion information and the first reference frame in the first portion of the decoder-side frame cache;
combining the motion-compensated prediction values with the residual values; and
performing deblock filtering.

4. The method of claim 1 wherein the first reference frame in the first portion of the decoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the decoder-side frame cache is updated to store a current frame of the plural frames for use in decoding a next frame of the plural frames.

5. The method of claim 1 wherein the encoded data is received over a real-time transport protocol channel, and wherein the loss detection signal is transmitted over a real-time control protocol channel.

6. The method of claim 1 wherein the detecting the loss of encoded data for one of the plural frames comprises discovering a gap in sequence numbers for plural packets due to packet loss or corruption of the encoded data.

7. The method of claim 1 further comprising repeating the transmitting of a loss detection signal until receipt of the encoded data for the new frame.

8. The method of claim 1 wherein the loss detection signal includes a timestamp and sequence number for a packet subject to the loss of encoded data.

9. The method of claim 1 wherein the loss detection signal includes a list of packets subject to the loss of encoded data.

10. The method of claim 1 wherein the loss detection signal identifies one or more slices subject to the loss of encoded data.

11. The method of claim 1 wherein the loss detection signal includes information that is usable by the video encoder to determine which stored reference frame to use among multiple reference frames stored at the video encoder, and wherein the encoded data for the new frame includes a frame header for the new frame, the frame header for the new frame including information that indicates to use the first reference frame in the decoding of the new frame.

12. In a computing system that implements a video encoder, a method of encoding plural frames of a video sequence, the method comprising:
encoding a first frame of the plural frames to produce encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;
storing, for use as a first reference frame, the first frame in a first portion of an encoder-side frame cache that corresponds to a first portion of a decoder-side frame cache at a video decoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;
encoding a second frame of the plural frames to produce encoded data for the second frame, the second frame following the first frame in the video sequence;
storing, for use as a second reference frame, the second frame in a second portion of the encoder-side frame cache different than the first portion of the encoder-side frame cache;
receiving a loss detection signal that indicates loss of encoded data for one of the plural frames; and
in response to the loss detection signal, encoding a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache to produce encoded data for the new frame, the new frame including different data than the one of the plural frames for which the loss detection signal indicates loss of encoded data, the new frame being decodable by the video decoder using the encoded data for the new frame and the first reference frame stored in the first portion of the decoder-side frame cache.

13. The method of claim 12 wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in the second portion of the encoder-side frame cache.

14. The method of claim 12 wherein the encoding the new frame includes:
performing motion compensation to produce motion-compensated prediction values using motion information and the first reference frame in the first portion of the encoder-side frame cache;
determining residual values using the motion-compensated prediction values;
frequency transforming the residual values to produce transform coefficients;
quantizing the transform coefficients to produce quantized transform coefficients; and
arithmetic coding the quantized transform coefficients and side information.

15. The method of claim 12 wherein the first reference frame in the first portion of the encoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the encoder-side frame cache is updated to store a current frame of the plural fr

Assignees

• Lin Chih-Lung
• Xia Minghui
• Hsu Pohsiang
• Regunathan Shankar
• Holcomb Thomas W
• Microsoft Technology Licensing Llc

Inventors

• Lin Chih-Lung
• Xia Minghui
• Hsu Pohsiang
• Regunathan Shankar
• Holcomb Thomas W

Classifications

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/00933 Primary

  Electricity · mapped topic

• H04N19/89 Primary

  involving methods or arrangements for detection of transmission errors at the decoder · CPC title

• H04N19/61

  in combination with predictive coding · CPC title

• H04N19/166

  concerning the amount of transmission errors, e.g. bit error rate [BER] · CPC title