Reference selection for video interpolation or extrapolation

Patent metadata

Field	Value
Publication number	US-9426414-B2
Application number	US-11171808-A
Country	US
Kind code	B2
Filing date	Apr 29, 2008
Priority date	Dec 10, 2007
Publication date	Aug 23, 2016
Grant date	Aug 23, 2016

Abstract

This disclosure describes selection of reference video units to be used for interpolation or extrapolation of a video unit, such as a video frame. A decoder may apply a quality-focused mode to select a reference frame based on quality criteria. The quality criteria may indicate a level of quality likely to be produced by a reference frame. If no reference frames satisfy the quality criteria, interpolation or extrapolation may be disabled. Display of an interpolated or extrapolated frame may be selectively enabled based on a quality analysis. A decoder may apply a resource-focused frame interpolation mode to enable or disable frame interpolation or extrapolation for some frames based on power and quality considerations. In one mode, frame interpolation may be disabled to conserve power when reference frames are not likely to produce satisfactory quality. In another mode, the threshold may be adjustable as a function of power saving requirements.

First claim

The invention claimed is:

1. A method comprising:
decoding, by a video decoding device, a plurality of candidate reference video units;
determining, by the video decoding device, to enable generation of an additional video unit between two consecutive video units of the plurality of candidate reference video units, wherein the plurality of candidate reference video units includes at least one candidate reference video unit in addition to the two consecutive video units;
in response to the determination to enable generation of the additional video unit, analyzing, by the video decoding device, at least one characteristic specified in a respective syntax element of each of the candidate reference video units, wherein the at least one characteristic includes a characteristic indicating a quality of a corresponding one of the candidate reference video units and indicating a likely quality of a generated video unit as generated from the corresponding one of the candidate reference video units prior to initiation of a process to generate the additional video unit from the corresponding one of the candidate reference video units; and
selecting, by the video decoding device, one or more of the candidate reference video units as a reference video unit for interpolation or extrapolation of the additional video unit based at least in part on the analysis, wherein at least one of the selected candidate reference video units is different than the two consecutive video units between which the additional video unit is to be generated.

2. The method of claim 1 , wherein each of the candidate reference video units is a candidate reference video frame and the additional video unit is an additional video frame.

3. The method of claim 1 , wherein analyzing the at least one characteristic includes analyzing a syntax element specifying a quantization parameter (QP) or a syntax element specifying coded block pattern (CBP) values of one or more of the candidate reference video units.

4. The method of claim 1 , wherein analyzing the at least one characteristic further includes detecting a video unit loss in one or more of the candidate reference video units, and wherein selecting one or more of the candidate reference video units as a reference video unit includes selecting one or more of the candidate reference video units based on a quality of error concealment.

5. The method of claim 1 , wherein analyzing the at least one characteristic further includes analyzing one or more objective visual quality metrics of one or more of the candidate reference video units.

6. The method of claim 1 , wherein selecting one or more of the candidate reference video units as a reference video unit comprises selecting one or more candidate reference video units for which the quality level satisfies at least one threshold.

7. The method of claim 6 , further comprising ranking, based on the quality level, at least some of the candidate reference video units that satisfy the at least one threshold, and selecting one or more of the candidate reference video units that are most highly ranked.

8. The method of claim 6 , further comprising adjusting the at least one threshold based on a level of available power resources.

9. The method of claim 1 , the method further comprising interpolating or extrapolating the additional video unit based on the analysis.

10. The method of claim 1 , wherein the at least one characteristic further comprises a second characteristic indicating a temporal distance of one or more of the candidate reference video units from the additional video unit.

11. The method of claim 10 , further comprising:
detecting a delay-sensitive video application; and
selecting the one or more of the candidate reference video units based at least in part on the temporal distance of one or more of the candidate reference video units from the additional video unit when the delay-sensitive video application is detected.

12. The method of claim 10 , further comprising selecting the one or more of the candidate reference video units based at least in part on the temporal distance of one or more of the candidate reference video units from the additional video unit.

13. The method of claim 1 , wherein the at least one characteristic includes a quantization parameter (QP) and a characteristic indicating a temporal distance of one or more of the candidate reference video units from the additional video unit.

14. A device comprising:
a video decoder that decodes a plurality of candidate reference video units;
an analysis unit that determines to enable generation of an additional video unit between two consecutive video units of the plurality of candidate reference video units, wherein the plurality of candidate reference video units includes at least one candidate reference video unit in addition to the two consecutive video units, and, in response to the determination to enable generation of the additional video unit, analyzes at least one characteristic specified in a respective syntax element of each of the candidate reference video units, wherein the at least one characteristic includes a characteristic indicating a quality of a corresponding one of the candidate reference video units and indicating a likely quality of a generated video unit as generated from the corresponding one of the candidate reference video units prior to initiation of a process to generate the additional video unit from the corresponding one of the candidate reference video units; and
a selection unit that selects one or more of the candidate reference video units as a reference video unit for interpolation or extrapolation of the additional video unit based at least in part on the analysis, wherein at least one of the selected candidate reference video units is different than the two consecutive video units between which the additional video unit is to be generated.

15. The device of claim 14 , wherein each of the candidate reference video units is a candidate reference video frame and the additional video unit is an additional video frame.

16. The device of claim 14 , wherein the analysis unit analyzes at least one of a syntax element specifying a quantization parameter (QP) or a syntax element specifying coded block pattern (CBP) values of one or more of the candidate reference video units.

17. The device of claim 14 , wherein the analysis unit detects video unit loss in one or more of the candidate reference video units, and the selection unit selects one or more of the candidate reference video units based on a quality of error concealment.

18. The device of claim 14 , wherein the analysis unit analyzes one or more objective visual quality metrics of one or more of the candidate reference video units.

19. The device of claim 14 , wherein the selection unit selects one or more candidate reference video units for which the quality level satisfies at least one threshold.

20. The device of claim 19 , wherein the analysis unit ranks, based on the quality level, at least some of the candidate reference video units that satisfy the at least one threshold, and the selection unit selects one or more of the candidate reference video units that are most highly ranked.

21. The device of claim 19 , further comprising an adjustment unit that adjusts the at least one threshold based on a level of available power resources.

22. The device of claim 14 , the device further comprising a substitution unit that interpolates or extrapolates the additional video unit ba

Assignees

• Dane Gokce
• El-Maleh Khaled Helmi
• Dai Min
• Teng Chia-Yuan
• Qualcomm Inc

Inventors

• Dane Gokce
• El-Maleh Khaled Helmi
• Dai Min
• Teng Chia-Yuan

Classifications

• H04N19/105

  Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title

• H04N19/573

  Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction · CPC title

• H04N19/156

  Availability of hardware or computational resources, e.g. encoding based on power-saving criteria · CPC title

• H04N7/0132 Primary

  the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction · CPC title