Content-adaptive background skipping for region-of-interest video coding

Patent metadata

Field	Value
Publication number	US-9667980-B2
Application number	US-20040705-A
Country	US
Kind code	B2
Filing date	Aug 9, 2005
Priority date	Mar 1, 2005
Publication date	May 30, 2017
Grant date	May 30, 2017

Abstract

The disclosure is directed to techniques for content-adaptive background skipping for region-of-interest (ROI) video coding. The techniques may be useful in video telephony (VT) applications such as video streaming and videoconferencing, and especially useful in low bit-rate wireless communication applications, such as mobile VT. The disclosed techniques analyze content information of a video frame to dynamically determine whether to skip a non-ROI area within the frame. For example, the skipping determination may be based on content activity, such as ROI shape deformation, ROI motion, non-ROI motion, non-ROI texture complexity, and accumulated distortion due to non-ROI skipping. The skip determination may operate in conjunction with either frame-level or macroblock-level bit allocation.

First claim

The invention claimed is:

1. A method for encoding video at an encoding module, comprising:
obtaining a video frame;
encoding a region of interest (ROI) and a non-ROI area within the video frame by allocating bits between the ROI and the non-ROI area;
determining content activity in the ROI;
determining content activity in the non-ROI area; and
determining whether to skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI exceeds a first threshold,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.

2. The method of claim 1 , wherein the content activity of the ROI includes shape deformation and motion in the ROI.

3. The method of claim 1 , wherein the content activity of the non-ROI area includes motion and texture complexity in the non-ROI area.

4. The method of claim 1 , further comprising allocating encoding bits between the ROI and the non-ROI area when encoding of the non-ROI area is not skipped.

5. The method of claim 4 , wherein allocating encoding bits includes allocating the encoding bits based on a frame-level bit budget, the method further comprising adjusting the frame-level bit budget based on historical and predictive skipping statistics for non-ROI areas of the video frame and other video frames.

6. The method of claim 5 , further comprising applying a Bayesian model to the historical skipping statistics to obtain the predictive skipping statistics.

7. The method of claim 4 , wherein allocating encoding bits includes allocating encoding bits in a ρ-domain at a macroblock level within the video frame.

8. The method of claim 4 , wherein allocating encoding bits includes allocating the encoding bits based on a ρ-domain bit budget.

9. The method of claim 8 , wherein allocating encoding bits includes applying a bit allocation scheme based on a rate control strategy that determines the ρ-domain bit budget based on a texture complexity of the ROI and non-ROI areas within the frame.

10. The method of claim 8 , wherein allocating encoding bits includes applying a bit allocation scheme based on a rate control strategy that estimates future skipping events based on statistics and patterns of non-ROI skipping history over previous frames within a video sequence, and then determines the ρ-domain bit budget based on the estimation.

11. A device comprising:
a region-of-interest mapper that generates a definition of a region of interest (ROI) within a video frame;
a video encoder that encodes the ROI and a non-ROI area within the video frame by allocating bits between the ROI and the non-ROI area; and
a skipping module that determines:
content activity in the ROI;
content activity in the non-ROI area; and
whether the video encoder will skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,

wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI exceeds a first threshold,
wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.

12. The device of claim 11 , wherein the content activity of the ROI includes shape deformation and motion in the ROI.

13. The device of claim 11 , wherein the content activity of the non-ROI area includes motion and texture complexity in the non-ROI area.

14. The device of claim 11 , further comprising a bit allocation module that allocates encoding bits between the ROI and the non-ROI area when encoding of the non-ROI area is not skipped.

15. The device of claim 14 , wherein the bit allocation module allocates the encoding bits based on a frame-level bit budget, the device further comprising a budget adjustment module that adjusts the frame-level bit budget based on historical and predictive skipping statistics for non-ROI areas of the video frame and other video frames.

16. The device of claim 15 , wherein the budget adjustment module applies a Bayesian model to the historical skipping statistics to obtain the predictive skipping statistics.

17. The device of claim 14 , wherein the bit allocation module allocates encoding bits in a ρ-domain at the macroblock level within the video frame.

18. The device of claim 14 , wherein the bit allocation module allocates the encoding bits based on a ρ-domain bit budget.

19. The device of claim 18 , wherein the bit allocation module applies a bit allocation scheme based on a rate control strategy that determines the ρ-domain budget based on a texture complexity of the ROI and non-ROI areas within the frame.

20. The device of claim 18 , wherein the bit allocation module applies a bit allocation scheme based on a rate control strategy that estimates future skipping events based on statistics and patterns of non-ROI skipping history over previous frames within a video sequence, and then determines the ρ-domain bit budget based on the estimation.

21. A non-transitory computer-readable medium encoded with a computer program comprising instructions to cause a processor to:
obtain a video frame;
encode a region of interest (ROI) and a non-ROI area within the video frame by allocating bits between the ROI and non-ROI area;
determine content activity in the ROI;
determine content activity in the non-ROI area; and
determine whether to skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI exceeds a first threshold,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.

Assignees

• Wang Haohong
• El-Maleh Khaled Helmi
• Liang Yi
• Qualcomm Inc

Inventors

• Wang Haohong
• El-Maleh Khaled Helmi
• Liang Yi

Classifications

• H04N19/177

  the unit being a group of pictures [GOP] · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title

• H04N19/587

  involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence · CPC title

• H04N19/115

  Selection of the code volume for a coding unit prior to coding · CPC title

• H04N19/126

  Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers · CPC title