Offloading video coding processes to hardware for better density-quality tradeoffs

Patent metadata

Field	Value
Publication number	US-12101475-B2
Application number	US-202017127544-A
Country	US
Kind code	B2
Filing date	Dec 18, 2020
Priority date	Dec 18, 2020
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Techniques related to distributing the video encoding processing of an input video across hardware and software systems. Such techniques include evaluating the content of the video and determine whether or the encoding operation is best to be done on the hardware system only, software system only or a hybrid hardware and software system.

First claim

What is claimed is:

1. A system for video coding comprising:
a memory to store at least a portion of input video for coding; and
one or more processors coupled to the memory, the one or more processors to:
determine a hardware operation is to be applied to encode a first picture of the input video, in response to determining that a first target quality of the first picture is attainable by hardware only, or that the first picture is a non-reference picture in a highest temporal layer of a hierarchical group of pictures structure;
determine a software operation or a hybrid hardware-software operation is to be applied to encode the first picture, in response to determining that the first target quality of the first picture is unattainable by hardware only, and that the first picture is not the non-reference picture in the highest temporal layer; and
generate an output bitstream by (1) encoding the first picture via a function call to hardware in response to determining that the hardware operation is to be applied, or (2) encoding the first picture via one or more of software and the function call to the hardware in response to determining that the software operation or the hybrid hardware-software operation is to be applied.

2. The system of claim 1 , wherein the one or more processors are to:
determine the first target quality of the first picture;
determine a quality level attainable using hardware by accessing a table; and
compare the first target quality of the first picture and the quality level attainable using hardware determined from the table.

3. The system of claim 1 , wherein the one or more processors are to:
encode a second picture using a multi-pass encode, in response to determining that multi-pass encode is required to meet a second quality target of the second picture, wherein the multi-pass encode comprises a first pass using the hardware and a subsequent pass at least partially performed in software.

4. The system of claim 3 , wherein the one or more processors are to:
perform single pass encode of the second picture comprising software encode up to a particular encode stage and hardware encode for one or more subsequent stages to the particular encode stage, in response to determining that multi-pass encode is not required to meet the second quality target.

5. The system of claim 4 , wherein the particular encode stage comprises a mode decision stage and the one or more subsequent stages comprise a reconstruction stage, a filtering stage, and an entropy coding stage.

6. The system of claim 1 , wherein the hybrid hardware-software operation comprises one or more first encode operations to be performed via hardware, and one or more second encode operations to be performed via software.

7. The system of claim 6 , wherein:
the one or more first encode operations comprise one or more of motion compensated temporal filtering, global motion estimation, flat picture analysis, spatial analysis, open loop intra search, film grain synthesis, picture based interpolation, picture reconstruction, loop filtering, and entropy coding; and
the one or more second encode operations comprise mode decision.

8. The system of claim 1 , wherein:
the hardware comprises one of fixed function circuitry or an application-specific integrated circuit; and
encoding the first picture via the software comprises running a plurality of lines of instruction code via a general purpose device.

9. The system of claim 1 , wherein the one or more processors are to:
decode a received bitstream via a second function call to second hardware to generate the input video.

10. The system of claim 9 , wherein encoding the first picture via the software is based at least in part on metadata generated by the decoding of the received bitstream.

11. A method for video coding comprising:
receiving input video for coding;
determining a hardware operation is to be applied to encode a first picture of the input video, in response to determining that a first target quality of the first picture is attainable by hardware only or that the first picture is a non-reference picture in a highest temporal layer of a hierarchical group of pictures structure;
determining a software operation or a hybrid hardware-software operation is to be applied to encode the first picture, in response to determining that the first target quality of the first picture is unattainable by hardware only, and that the first picture is not the non-reference picture in the highest temporal layer; and
generating an output bitstream by (1) encoding the first picture via a function call to hardware in response to determining that the hardware operation is to be applied, or (2) encoding the first picture via one or more of software and the function call to the hardware in response to determining that the software operation or the hybrid hardware-software operation is to be applied.

12. The method of claim 11 , further comprising:
determining the first target quality of the first picture;
determining a quality level attainable using hardware by accessing a table; and
comparing the first target quality of the first picture and the quality level attainable using hardware determined from the table.

13. The method of claim 11 , further comprising:
encoding a second picture using multi-pass encoding, in response to determining that multi-pass encoding is required to meet a second quality target of the second picture, wherein the multi-pass encoding comprises performing a first pass using the hardware and performing a subsequent pass at least partially in software.

14. The method of claim 13 , further comprising:
performing single pass encode of the second picture comprising software encode up to a particular encode stage and hardware encode for one or more subsequent stages to the particular encode stage, in response to determining that multi-pass encode is not required to meet the second quality target.

15. The method of claim 14 , wherein the particular encode stage comprises a mode decision stage and the one or more subsequent stages comprise a reconstruction stage, a filtering stage, and an entropy coding stage.

16. The method of claim 11 , further comprising:
determining the hardware operation is to be applied to encode the first picture, in response to determining that the first picture is in a second highest temporal layer of the hierarchical group of pictures structure.

17. The method of claim 11 , wherein:
the hybrid hardware-software operation comprises one or more first encode operations to be performed via hardware, and one or more second encode operations to be performed via software;
the one or more first encode operations comprise one or more of motion compensated temporal filtering, global motion estimation, flat picture analysis, spatial analysis, open loop intra search, film grain synthesis, picture based interpolation, picture reconstruction, loop filtering, and entropy coding; and
the one or more second encode operations comprise mode decision.

18. At least one machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform video coding by:
receiving input video for coding;
determining a hardware operation is to be applied to encode a first picture of the input video in response to determining that a first target quality of the first picture is attainable by hardware only or that the first picture is a non-reference picture in a highest temporal layer of a hierarchical group of pict

Assignees

• Intel Corp

Inventors

• Ganesh Brinda
• Jain Nilesh
• Mohan Sumit
• Kossentini Faouzi
• Boyce Jill
• Holland James
• Lei Zhijun
• Nouira Chekib
• Ben Amara Foued
• Tmar Hassene
• Possos Sebastian
• Hurst Craig

Classifications

• H04N19/154

  Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/164

  Feedback from the receiver or from the transmission channel · CPC title

• H04N19/42 Primary

  characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) · CPC title

• H04N19/82

  involving filtering within a prediction loop · CPC title