Adaptive upsampling for multi-layer video coding

Patent metadata

Field	Value
Publication number	US-10218971-B2
Application number	US-201314431978-A
Country	US
Kind code	B2
Filing date	Sep 27, 2013
Priority date	Sep 28, 2012
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

Systems, methods, and instrumentalities are disclosed for adaptive upsampling for multi-layer video coding. A method of communicating video data may involve applying an upsampling filter to a video sequence to create encoded enhancement layer pictures. The upsampling filter may be applied at a sequence level of the video sequence to create the enhancement layer bitstream. The upsampling filter may be selected from a plurality of candidate upsampling filters, for example, by determining whether knowledge of a category related to the video sequence exists and selecting a candidate upsampling filter that is designed for the category related to the video sequence. Upsampling filter information may be encoded. The encoded upsampling information may comprise a plurality of coefficients of the upsampling filter. The encoded upsampling filter information and the encoded enhancement layer pictures may be sent in an output video bitstream. The method may be performed, for example, by an encoder.

First claim

What is claimed:

1. A method for communicating video data, the method comprising:
selecting an upsampling filter for a video sequence to create enhancement layer pictures by:
applying a default upsampling filter to a base layer picture to obtain a processed base layer picture;
identifying blocks of the processed base layer picture that are selected for interlayer prediction of an enhancement layer frame; and
selecting one of the default upsampling filter and the plurality of candidate upsampling filters based on a comparison of block distortion measurements on the identified blocks associated with each of the default upsampling filter and the plurality of candidate upsampling filters;

encoding upsampling filter information, wherein the upsampling filter information comprises a plurality of coefficients of the selected upsampling filter; and
sending the encoded upsampling filter information and the enhancement layer pictures in an output video bitstream.

2. The method of claim 1 , wherein the selected upsampling filter is applied at a sequence level of the video sequence to create the enhancement layer bitstream.

3. The method of claim 1 , further comprising selecting a second upsampling filter from a plurality of candidate upsampling filters by:
determining whether knowledge of a content of the video sequence exists; and
upon determining that knowledge of the content of the video sequence exists, selecting a candidate upsampling filter that is designed for the content of the video sequence.

4. The method of claim 1 , further comprising selecting a second upsampling filter from a plurality of candidate upsampling filters by:
determining whether knowledge of a category related to the video sequence exists; and
upon determining that knowledge of the category related to the video sequence exists, selecting a candidate upsampling filter that is designed for the category related to the video sequence.

5. The method of claim 1 , wherein the selected upsampling filter is applied at a picture or slice level of the video sequence.

6. The method of claim 5 , further comprising selecting a second upsampling filter from a plurality of candidate upsampling filters by:
performing multi-pass encoding on an enhancement layer picture of the video sequence utilizing the plurality of candidate upsampling filters; and
selecting the second upsampling filter from the plurality of candidate upsampling filters based on the performance of the second upsampling filter during the multi-pass encoding.

7. The method of claim 6 , wherein the performance of the second upsampling filter is evaluated using a rate distortion cost.

8. The method of claim 1 , wherein selecting one of the default upsampling filter and the plurality of candidate upsampling filters based on a comparison of block distortion measurements comprises selecting one of the default upsampling filter and the plurality of candidate upsampling filters with the lowest associated distortion.

9. The method of claim 5 , further comprising selecting a second upsampling filter by:
performing multi-pass encoding on an enhancement layer picture of the video sequence for each of a first plurality of candidate upsampling filters; and
selecting one of the first plurality of candidate upsampling filters corresponding to a coding pass that best satisfies a criterion.

10. The method of claim 1 , further comprising:
for a previously received video sequence, generating a set of training data comprising a reconstructed base layer picture and an original enhancement layer input picture of the previously received video sequence; and
training the upsampling filter using a least squared method that uses the set of training data of the previously received video sequence.

11. A system comprising executable instruction that cause the system to perform operations comprising:
selecting an upsampling filter for a video sequence to create enhancement layer pictures by:
applying a default upsampling filter to a base layer picture to obtain a processed base layer picture;
identifying blocks of the processed base layer picture that are selected for interlayer prediction of an enhancement layer frame; and
selecting one of the default upsampling filter and the plurality of candidate upsampling filters based on a comparison of block distortion measurements on the identified blocks associated with each of the default upsampling filter and the plurality of candidate upsampling filters;

encoding upsampling filter information, the upsampling filter information comprising a plurality of coefficients of the selected upsampling filter; and
sending the encoded upsampling filter information and the enhancement layer pictures in an output video bitstream.

12. The system of claim 11 , wherein the selected upsampling filter is applied at a sequence level of the video sequence to create the enhancement layer bitstream.

13. The system of claim 11 , further comprising executable instruction that cause the system to perform further operations comprising:
selecting a second upsampling filter from a plurality of candidate upsampling filters by:
determining whether knowledge of a content of the video sequence exists; and
upon determining that knowledge of the content of the video sequence exists, selecting a candidate upsampling filter that is designed for the content of the video sequence.

14. The system of claim 11 , further comprising executable instruction that cause the system to perform further operations comprising:
selecting a second upsampling filter from a plurality of candidate upsampling filters by:
determining whether knowledge of a category related to the video sequence exists; and
upon determining that knowledge of the category related to the video sequence exists, selecting a candidate upsampling filter that is designed for the category related to the video sequence.

15. The system of claim 11 , wherein the selected upsampling filter is applied at a picture or slice level of the video sequence.

16. The system of claim 15 , further comprising executable instruction that cause the system to perform further operations comprising:
selecting a second upsampling filter from a plurality of candidate upsampling filters by:
performing multi-pass encoding on an enhancement layer picture of the video sequence utilizing the plurality of candidate upsampling filters; and
selecting the second upsampling filter from the plurality of candidate upsampling filters based on the performance of the second upsampling filter during the multi-pass encoding.

17. The system of claim 16 , wherein the performance of the second upsampling filter is evaluated using a rate distortion cost.

18. The system of claim 11 , wherein selecting one of the default upsampling filter and the plurality of candidate upsampling filters based on a comparison of block distortion measurements comprises selecting one of the default upsampling filter and the plurality of candidate upsampling filters with the lowest associated distortion.

19. The system of claim 5 , further comprising executable instruction that cause the system to perform further operations comprising:
selecting a second upsampling filter by:
performing multi-pass encoding on an enhancement layer picture of the video sequence for each of a first plurality of candidate upsampling filters; and
selecting one of the first plurality of candidate upsampling filters corresponding to a coding pass that best satisfies a criterion.

20. The system of claim 1

Assignees

• Vid Scale Inc

Inventors

• Dong Jie
• Luo Enming
• He Yuwen
• Ye Yan

Classifications

• H04N19/192

  the adaptation method, adaptation tool or adaptation type being iterative or recursive · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/179

  the unit being a scene or a shot · CPC title

• H04N19/147

  according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title