Apparatus and method with video processing using neural network

Patent metadata

Field	Value
Publication number	US-12563196-B2
Application number	US-202318343916-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jan 17, 2023
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

An apparatus with video processing includes: one or more processors configured to: generate a syntax element processable by a target standard codec by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to a neural network and compressing the plurality of frames, and generate a bitstream by performing entropy encoding on the syntax element.

First claim

What is claimed is:

1 . An apparatus with video processing, the apparatus comprising:
one or more processors configured to:
generate syntax elements processable by a target decoder of a target standard codec by selecting one of outputs generated by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks, and
generate a bitstream by performing entropy encoding on the syntax elements.

2 . The apparatus of claim 1 , wherein the syntax elements comprise a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.

3 . The apparatus of claim 1 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.

4 . The apparatus of claim 1 , wherein, for the generating of the pre-decoded reference image, the one or more processors are further configured to:
generate decoded syntax elements by performing entropy decoding on the bitstream, and generate the pre-decoded reference image by decompressing the decoded syntax elements.

5 . The apparatus of claim 4 , wherein the decoded syntax elements are decodable by the target decoder comprising a decoder of a standard codec.

6 . The apparatus of claim 1 , wherein the neural networks comprise a first neural network and a second neural network.

7 . The apparatus of claim 6 , wherein, for the generating of the bitstream, the one or more processors are further configured to:
select one from an output of the first neural network and an output of the second neural network, and perform entropy encoding on the selected output.

8 . The apparatus of claim 6 , wherein, for the generating of the syntax elements, the one or more processors are further configured to perform either one or both of:
intra-prediction through the first neural network; and inter-prediction through the second neural network.

9 . The apparatus of claim 7 , wherein, for the generating of the syntax elements, the one or more processors are further configured to:
partition the plurality of frames into a plurality of blocks through the first neural network, and perform motion estimation and compensation by inputting the plurality of blocks to the second neural network.

10 . The apparatus of claim 1 , wherein, for the generating of the syntax elements, the one or more processors are further configured to adjust the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.

11 . A processor-implemented method with video processing, the method comprising:
generating syntax elements processable by a target decoder of a target standard codec by selecting one of outputs generated by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks; and generating a bitstream by performing entropy encoding on the syntax elements.

12 . The method of claim 11 , wherein the syntax elements comprise a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.

13 . The method of claim 11 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.

14 . The method of claim 11 , wherein the generating of the pre-decoded reference image comprises:
generating decoded syntax elements by performing entropy decoding on the bitstream; and generating the pre-decoded reference image by decompressing the decoded syntax elements.

15 . The method of claim 14 , wherein the decoded syntax elements are decodable by the target decoder comprising a decoder of a standard codec.

16 . The method of claim 11 , wherein the neural networks comprise a first neural network and a second neural network.

17 . The method of claim 16 , wherein the generating of the bitstream comprises:
selecting one from an output of the first neural network and an output of the second neural network; and performing entropy encoding on the selected output.

18 . The method of claim 16 , wherein the generating of the syntax elements comprises either one or both of:
performing intra-prediction through the first neural network; and performing inter-prediction through the second neural network.

19 . The method of claim 17 , wherein the generating of the syntax elements comprise:
partitioning the plurality of frames into a plurality of blocks through the first neural network; and performing motion estimation and compensation by inputting the plurality of blocks to the second neural network.

20 . The method of claim 11 , wherein the generating of the syntax elements comprises adjusting the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Seungeon
• Lee Wonhee
• Jung Kyungboo
• CHOI Woosuk
• Sung Young Hun
• Oh Dokwan

Classifications

• H04N19/70 Primary

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• H04N19/91 Primary

  Entropy coding, e.g. variable length coding [VLC] or arithmetic coding · CPC title

• H04N19/51

  Motion estimation or motion compensation · CPC title

• G06N3/045

  Combinations of networks · CPC title

• H04N19/105

  Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction · CPC title