Adaptive filtering mechanism to remove encoding artifacts in video data

Patent metadata

Field	Value
Publication number	US-9118932-B2
Application number	US-201313917669-A
Country	US
Kind code	B2
Filing date	Jun 14, 2013
Priority date	Jun 14, 2013
Publication date	Aug 25, 2015
Grant date	Aug 25, 2015

Abstract

A method includes determining, through a processor and/or a hardware engine, edge pixels and flat pixels of a video frame of a video sequence during decoding thereof or post-processing associated with the decoding based on a predetermined threshold, and quantifying spatial correlation of pixels of the video frame around edges thereof to estimate strength of ringing artifacts and spatial and temporal persistence thereof across the video frame and across video frames of the video sequence. The method also includes adaptively and spatially filtering the pixels around the edges of the video frame, adaptively and temporally filtering the video frame, and blending an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

First claim

What is claimed is:

1. A method comprising:
determining, through at least one of a processor of a data processing device communicatively coupled to a memory and a hardware engine, edge pixels and flat pixels of a video frame of a video sequence during one of decoding thereof and post-processing associated with the decoding based on a predetermined threshold, the edge pixels being associated with at least one edge of the video frame around which there is a change in intensity level above a threshold value thereof and the flat pixels being associated with at least one area of the video frame around which the change in the intensity level is below the threshold value;
quantifying, through the at least one of the processor and the hardware engine, spatial correlation of pixels of the video frame around the at least one edge to estimate a strength of ringing artifacts therein, persistence of the ringing artifacts within the video frame and temporal persistence thereof across the video frame and another video frame of the video sequence based on the determined edge pixels and the flat pixels and a raw output of the decoding;
adaptively and spatially filtering, through an edge preserving filter implemented through at least one of: executing instructions on the processor and the hardware engine, the pixels around the at least one edge of the video frame based on metadata associated with the video sequence, the estimated strength of the ringing artifacts, the persistence of the ringing artifacts within the video frame and the temporal persistence of the ringing artifacts across the video frame and the another video frame;
adaptively and temporally filtering, through the at least one of the processor and the hardware engine, the video frame to mitigate artifacts resulting from a cumulative effect of compression during encoding of the video sequence; and
blending, through the at least one of the processor and the hardware engine, an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

2. The method of claim 1 , wherein determining the edge pixels and the flat pixels further comprises at least one of:
estimating the at least one edge of the video frame;
marking at least one pixel surrounding an edge pixel also as an edge pixel to widen a contour of the at least one edge;
computing a range for each sub-block of the video frame as a difference between a maximum and a minimum of intensity levels of pixels therewithin;
comparing the range for the each sub-block to another predetermined threshold to determine the flat pixels and the edge pixels therewithin; and
preserving the edge pixels within the each sub-block.

3. The method of claim 2 , further comprising preventing filtering of the each sub-block when a flat pixel percentage and an edge pixel percentage thereof are respectively above a first threshold value and a second threshold value.

4. The method of claim 2 , comprising computing an approximate gradient of an intensity level function of the video frame using a Sobel operator as part of the estimation of the at least one edge thereof.

5. The method of claim 1 , comprising at least one of:
implementing a bilateral filter having a spatial filter component and a range filter component as the edge preserving filter; and
implementing a first-order Infinite Impulse Response (IIR) filter as a component performing the adaptive temporal filtering, the IIR filter being configured to generate an output by recursively applying alpha-blending of the video frame with a previous temporally filtered video frame of the video sequence.

6. The method of claim 5 , further comprising adapting the bilateral filter through varying at least one of a deviation parameter associated with the spatial filter component and a radiometric deviation parameter associated with the range filter component.

7. The method of claim 6 , comprising implementing the spatial filter component and the range filter component as Gaussian functions.

8. A data processing device comprising:
a memory; and
a processor communicatively coupled to the memory, the processor being configured to execute instructions to:
determine edge pixels and flat pixels of a video frame of a video sequence during one of decoding thereof and post-processing associated with the decoding based on a predetermined threshold, the edge pixels being associated with at least one edge of the video frame around which there is a change in intensity level above a threshold value thereof and the flat pixels being associated with at least one area of the video frame around which the change in the intensity level is below the threshold value,
quantify spatial correlation of pixels of the video frame around the at least one edge to estimate a strength of ringing artifacts therein, persistence of the ringing artifacts within the video frame and temporal persistence thereof across the video frame and another video frame of the video sequence based on the determined edge pixels and the flat pixels and a raw output of the decoding,
adaptively and spatially filter, through an edge preserving filter, the pixels around the at least one edge of the video frame based on metadata associated with the video sequence, the estimated strength of the ringing artifacts, the persistence of the ringing artifacts within the video frame and the temporal persistence of the ringing artifacts across the video frame and the another video frame,
adaptively and temporally filter the video frame to mitigate artifacts resulting from a cumulative effect of compression during encoding of the video sequence, and
blend an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

9. The data processing device of claim 8 , wherein the processor is configured to execute instructions to determine the edge pixels and the flat pixels based on:
estimating the at least one edge of the video frame,
marking at least one pixel surrounding an edge pixel also as an edge pixel to widen a contour of the at least one edge thereof,
computing a range for each sub-block of the video frame as a difference between a maximum and a minimum of intensity levels of pixels therewithin,
comparing the range for the each sub-block to another predetermined threshold to determine the flat pixels and the edge pixels therewithin, and
preserving the edge pixels within the each sub-block.

10. The data processing device of claim 9 , wherein the processor is further configured to execute instructions to prevent filtering of the each sub-block when a flat pixel percentage and an edge pixel percentage thereof are respectively above a first threshold value and a second threshold value.

11. The data processing device of claim 9 , wherein the processor is configured to execute instructions to compute an approximate gradient of an intensity level function of the video frame using a Sobel operator as part of the estimation of the at least one edge thereof.

12. The data processing device of claim 8 , wherein the processor is configured to execute instructions associated with at least one of:
a bilateral filter having a spatial filter component and a range filter component as the edge preserving filter, and
a first-order IIR filter as a component performing the adaptive temporal filtering, the IIR filter being configured to generate an output by

Assignees

• Nvidia Corp

Inventors

• Avadhanam Niranjan
• Boddeti Ravi Kumar

Classifications

• H04N19/117

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title

• H04N19/14

  Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence) · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/86 Primary

  involving reduction of coding artifacts, e.g. of blockiness · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title