Hardware codec accelerators for high-performance video encoding

Patent metadata

Field	Value
Publication number	US-12170757-B2
Application number	US-202117451972-A
Country	US
Kind code	B2
Filing date	Oct 22, 2021
Priority date	Sep 2, 2021
Publication date	Dec 17, 2024
Grant date	Dec 17, 2024

Abstract

Disclosed are apparatuses, systems, and techniques for real-time codec encoding of video files using hardware-assisted accelerators that utilize a combination of parallel and sequential processing, in which at least a part of intra-frame block prediction is performed with parallel processing.

First claim

What is claimed is:

1. A system comprising:
a memory device to store source pixel values of an image frame; and
one or more circuits communicatively coupled to the memory device, the one or more circuits to:
segment the image frame into a first plurality of partitions of a first size;
associate, with each of the first plurality of partitions, a respective predicted pixel block (PB) of a first plurality of predicted PBs;
identify, using the first plurality of predicted PBs, a reference pixel grid (RPG) that comprises pixel values of a subset of pixels of the first plurality of predicted PBs;
segment the image frame into a second plurality of partitions of a second size;
obtain, using the RPG identified using the first plurality of predicted PBs of the first size, associations of each of the second plurality of partitions with one or more predicted PBs of a second plurality of predicted PBs of the second size, wherein at least two of the associations are obtained in parallel;
select final predicted PBs from a plurality of candidate PBs, wherein the plurality of candidate PBs comprises the first plurality of predicted PBs and the second plurality of predicted PBs;
apply one or more transformations to the final predicted PBs to obtain reconstructed PBs; and
create a representation of the image frame using pixel values of the reconstructed PBs.

2. The system of claim 1 , wherein to select the final predicted PBs, the one or more circuits are to use a comparison of the pixel values of a respective plurality of predicted PBs with source pixel values of the image frame.

3. The system of claim 1 , wherein to obtain the associations, the one or more circuits are to:
identify a plurality of candidate predicted PBs, each of the plurality of candidate predicted PBs generated using a respective interpolation mode of a plurality of interpolation modes; and
select the one or more predicted PBs based on a comparison of pixel values of each of the plurality of candidate predicted PBs with source pixel values of the image frame and using a first cost function.

4. The system of claim 3 , wherein the final predicted PBs are selected using a second cost function different from the first cost function.

5. The system of claim 1 , wherein the plurality of candidate PBs further comprises a plurality of inter-frame reference PBs, wherein each of the plurality of inter-frame reference PBs corresponds to a reference PB in one or more reference image frames, wherein the final predicted PBs comprise one or more inter-frame reference PBs chosen from the plurality of inter-frame reference PBs, and wherein to create the representation of the image frame, the one or more circuits are to:
select an invertible discrete transformation for each of the chosen one or more inter-frame reference PBs, based on pixel values of a respective inter-frame reference PB.

6. The system of claim 1 , wherein to apply the one or more transformations to the final predicted PBs, the one or more circuits are to:
compute a residual difference PB between an individual final predicted PB of the final predicted PBs and a corresponding source PB;
apply an invertible discrete transformation to the computed residual difference PB to obtain a transformed residual difference PB; and
apply a non-invertible quantization transformation to the transformed residual difference PB to obtain a quantized residual difference PB.

7. The system of claim 6 , wherein to apply the one or more transformations to the final predicted PBs, the one or more circuits are further to:
apply a reverse quantization transformation to the quantized residual difference PB to obtain a reverse quantized residual difference PB; and
apply an inverse discrete transformation to the reverse quantized residual difference PB to obtain an individual reconstructed PB of the reconstructed PBs.

8. The system of claim 1 , wherein to create the representation of the image frame using the pixel values of the reconstructed PBs, the one or more circuits are to:
identify parameters of one or more filters to be applied by an image decoder to the reconstructed PBs; and
include the identified parameters of the one or more filters in the representation of the image frame.

9. A system comprising:
a memory device to store source pixel values of an image; and
one or more circuit groups communicatively coupled to the memory device, the one or more circuit groups comprising:
a first circuit group to:
identify, for each of a first plurality of partitions, one or more initial interpolation modes of a plurality of interpolation modes of a video codec encoder;
identify, using the one or more initial interpolation modes, a reference pixel grid (RPG) that comprises pixel values of a subset of pixels of the first plurality of partitions; and
associate, using the RPG, with each of a second plurality of partitions of the image, a selected interpolation mode of the plurality of interpolation modes of the video codec encoder; and

a second circuit group communicatively coupled to the first circuit group, the second circuit group to:
process the second plurality of partitions to obtain a plurality of reconstructed pixel blocks (PBs), wherein at least some reconstructed PBs of the plurality of reconstructed PBs are obtained using a respective selected interpolation mode associated with a respective partition of the second plurality of partitions; and
create a compressed representation of the image using the plurality of reconstructed PBs of the image.

10. The system of claim 9 , wherein partitions of the first plurality of partitions have a same size, wherein the second plurality of partitions comprises partitions of multiple sizes, and wherein to associate, with each of the second plurality of partitions of the image, the respective selected interpolation mode, the first circuit group is to:
chose i) the selected interpolation mode and ii) a size of the respective partition of the second plurality of partitions in view of a cost value that characterizes a difference between source pixels of the respective partition and pixels of a predicted PB generated using the respective selected interpolation mode.

11. The system of claim 9 , wherein the one or more circuit groups further comprise:
a motion estimation (ME) circuit group communicatively coupled to the first circuit group, the ME circuit group to:
identify, for each of a third plurality of partitions of the image, an inter-frame reference PB corresponding to a reference PB in one or more reference image frames;
and wherein at least some reconstructed PBs of the plurality of reconstructed PBs are obtained using the inter-frame reference PB for a respective partition of the third plurality of partitions.

12. A method comprising:
segmenting an image frame into a first plurality of partitions of a first size;
associating, with each of the first plurality of partitions, a respective predicted pixel block (PB) of a first plurality of predicted PBs;
identifying, using the first plurality of predicted PBs, a reference pixel grid (RPG) that comprises pixel values of a subset of pixels of the first plurality of predicted PBs;
segmenting the image frame into a second plurality of partitions of a second size;
obtain, using the RPG identified using the first plurality of predicted PBs, associations of each of the second plurality of partitions with one or more predicted PBs of a second plurality of predicted PBs, wherein at least two of the associations are obtained in parallel;
selecting final predicted PBs from a plurality of candidate PBs, wherein the plurality of candidate PBs comprises the first plurality of pr

Assignees

• Nvidia Corp

Inventors

• Srinivasan Ranga Ramanujam
• Chen Jianjun
• Zhang Dong
• Feng Wei
• He Xi

Classifications

• H04N19/12

  Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 · CPC title

• H04N19/159

  Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction · CPC title

• H04N19/61

  in combination with predictive coding · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/119 Primary

  Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks · CPC title