Method and system of video coding using content based metadata

Patent metadata

Field	Value
Publication number	US-11223831-B2
Application number	US-201916533607-A
Country	US
Kind code	B2
Filing date	Aug 6, 2019
Priority date	Jul 1, 2016
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

Techniques related to video coding using content based metadata.

First claim

What is claimed is:

1. A computer-implemented method of video coding comprising:
obtaining temporal display content information of image data of at least one area of non-compressed frames of a frame sequence to be rendered, wherein each area is less than an entire frame, wherein the temporal display content information comprises one of at least three available rendering classifications of dirty, scroll, or skip, and wherein the rendering classification is generated to render an image regardless of whether or not the frame sequence is to be provided to an encoder to be encoded;
obtaining pixel image data of the frames to separately encode the frame sequence; and
encoding the pixel image data to display the frame sequence at a device remote from a device rendering the frame sequence, the encoding comprising determining a prediction mode of at least one portion of at least one frame of the frame sequence and determining the prediction mode based on, at least in part, the temporal display content information.

2. The method of claim 1 wherein the temporal display content information comprises information that the at least one portion has at least one among three available options of:
dirty image data wherein the image data of the at least one portion is new image data of the frame relative to the image data on another frame in the frame sequence,
static image data wherein the image data of the at least one portion is the same content and position on the frame relative to the image data on another frame in the frame sequence, and
scrolled image data wherein the image data of the at least one portion has the same content but in a different position on the frame relative to the position of the image data on another frame in the frame sequence.

3. The method of claim 1 wherein the temporal display content information is provided to an encoder in the form of locations of regions each with a different region class and without dividing the regions into blocks; and the method comprising locating blocks of the frames formed at the encoder at one of the regions.

4. The method of claim 1 comprising providing the temporal display content information in a block granularity that matches one or more prediction unit partition sizes available for encoding the video sequence.

5. The method of claim 4 wherein the temporal display content information is provided to an encoder in a large block granularity and is treated as a dirty block when at least one sub-division block of the large block is considered a dirty block.

6. The method of claim 4 wherein the smallest block subdivision compatible with an encoder coding standard is the block granularity level of the temporal display content information.

7. The method of claim 1 comprising dividing the frames into blocks to form a block granularity of the temporal display content information wherein individual blocks are assigned one of region classes forming the temporal display content information; and
providing the block granularity temporal display content information to an encoder so that the encoder can use the block granularity temporal display content information to determine sub-divisions of blocks as prediction unit partitions.

8. The method of claim 7 comprising using encoder based prediction unit partitions rather than temporal display content information based partitions provided to or formed by the encoder for the same portion of the frame.

9. The method of claim 7 comprising providing blocks with prediction unit partitions based on temporal display content information for inter-prediction, intra-prediction and/or prediction mode selection.

10. The method of claim 7 comprising aggregating blocks assigned with the same region class of the temporal display content information to form a metadata-based modified block arrangement to be used by the encoder instead of a prediction unit partitioning of the encoder formed without using the temporal display content information.

11. The method of claim 1 comprising coding a skip block of the at least one portion as skip without subdividing the block and without determining a prediction mode for sub-divisions of the block.

12. The method of claim 1 comprising coding a scroll block of the at least one portion as inter-prediction and a motion vector that is the distance and direction of a scroll offset, and without determining predictions for sub-divisions of the scroll block and without coding a residual for the scroll block.

13. A method of video coding comprising:
obtaining temporal display content information of image data of at least one area of non-compressed frames of a frame sequence to be rendered, wherein each area is less than an entire frame, wherein the temporal display content information comprises one of at least three available rendering region classifications of dirty, scroll, or skip wherein the rendering classification is generated to render an image regardless of whether or not the frame sequence is to be provided to an encoder to be encoded;
obtaining pixel image data of the frames to separately encode the frame sequence to display the frame sequence on a device remote from a device to be used for the rendering; and
dividing the frames into blocks;
assigning one of the region classifications of the temporal display content information to individual blocks; and
providing block data of the blocks comprising the region classification of the blocks and to an encoder to determine prediction modes for the blocks using the region classifications and to compress the image data of the frames.

14. The method of claim 13 wherein the temporal display content information comprises information that the at least one portion has at least one data type of: dirty image data, static image data, and scrolled image data, and the method comprising determining or omitting prediction or residual generation to perform encoding of the portion and depending on the data type of data of the portion.

15. The method of claim 13 comprising providing the temporal display content information in a block granularity that matches one or more prediction unit partition sizes available for encoding the video sequence.

16. A computer-implemented system comprising:
at least one display;
at least one memory to store frames of a video sequence wherein individual frames have image data with pixels divided into blocks; and
at least one processor communicatively coupled to the at least one of the memory and display, and the at least one processor being arranged to operate by:
obtaining temporal display content information of image data of at least one area of non-compressed frames of a frame sequence to be rendered, wherein each area is less than an entire frame, wherein the temporal display content information comprises one of at least three available rendering classifications of dirty, scroll, or skip wherein the rendering classification is generated to render an image regardless of whether or not the frame sequence is provided to an encoder to be encoded;
obtaining pixel image data of the frames to separately encode the frame sequence; and
encoding the pixel image data to display the frame sequence at a device remote from a device to be used to perform the rendering, the encoding comprising determining a prediction mode of at least one portion of at least one frame of the frame sequence and determining the prediction mode based on, at least in part, the temporal display content information.

17. The system of claim 16 wherein the temporal display content information comprises information that the at least

Assignees

• Intel Corp

Inventors

• Noraz Frederic J
• Boyce Jill M
• Mohan Sumit

Classifications

• H04N19/176 Primary

  the region being a block, e.g. a macroblock · CPC title

• H04N19/102

  characterised by the element, parameter or selection affected or controlled by the adaptive coding · CPC title

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/139 Primary

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title

• H04N19/51

  Motion estimation or motion compensation · CPC title