Image processing apparatus using neural network and image processing method using the same

Patent metadata

Field	Value
Publication number	US-12598322-B2
Application number	US-202418654604-A
Country	US
Kind code	B2
Filing date	May 3, 2024
Priority date	Nov 15, 2023
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

An image encoding apparatus may include a memory storing one or more instructions; and a processor configured to execute the one or more instructions to: obtain an initial search position from a student neural network that has been trained through a knowledge distillation from a teacher neural network by inputting a current frame and a reference frame to the student neural network; and perform motion estimation based on the initial search position.

First claim

What is claimed is:

1 . An image encoding apparatus comprising:
a memory storing one or more instructions; and a processor configured to execute the one or more instructions to: obtain an initial search position from a student neural network that has been trained through a knowledge distillation from a teacher neural network by inputting a current frame and a reference frame to the student neural network; and perform motion estimation based on the initial search position, wherein the student neural network is trained by inputting a first frame and a second frame into the teacher neural network to obtain a first initial search position, inputting the first frame and the second frame into the student neural network to obtain a second initial search position, and training the student neural network based on a loss function using a difference between the first initial search position and the second initial search position.

2 . The image encoding apparatus of claim 1 , wherein the loss function comprises a multiplication of the difference between the first initial search position and the second initial search position by a loss weight.

3 . The image encoding apparatus of claim 2 , wherein the loss weight is determined based on bitrate increasing rate information acquired by the teacher neural network.

4 . The image encoding apparatus of claim 3 , wherein the bitrate increasing rate information is obtained by comparing an image compression result with the first initial search position applied and an image compression result without applying the first initial search position.

5 . The image encoding apparatus of claim 1 , wherein the processor is further configured to output a predicted frame by performing motion compensation based on result of the motion estimation.

6 . The image encoding apparatus of claim 1 , wherein the teacher neural network is located within the image encoding apparatus, or an external device located outside the image encoding apparatus.

7 . The image encoding apparatus of claim 1 , wherein there are a plurality of teacher neural networks comprising the teacher neural network, and
the student neural network is trained by inputting the first frame and the second frame into each of the plurality of teacher neural networks, obtaining a plurality of initial search positions from the plurality of teacher neural networks, inputting the first frame and the second frame into the student neural network, obtaining the initial search position from the student neural network, and train the student neural network based on the loss function using differences between the plurality of initial search positions from the plurality of teacher neural networks and the initial search position from the student neural network.

8 . The image encoding apparatus of claim 1 , wherein the loss function comprises a computation of a difference between the first initial search position and a ground truth label.

9 . An image encoding method comprising:
outputting an initial search position from a student neural network by inputting a current frame and a reference frame of an image into the student neural network that has been trained through a knowledge distillation from a teacher neural network; and performing motion estimation based on the initial search position, wherein the outputting of the initial search position comprises: obtaining a first initial search position from the teacher neural network by inputting a first frame and a second frame, into the teacher neural network; obtaining a second initial search position from the student neural network by inputting the first frame and the second frame into the student neural network; and training the student neural network based on a loss function using a difference between the first initial search position and the second initial search position.

10 . The image encoding method of claim 9 , wherein the loss function comprises a multiplication of the difference between the first initial search position and the second initial search position by a loss weight.

11 . The image encoding method of claim 10 , further comprising:
generating the loss weight based on bitrate increasing rate information acquired by the teacher neural network.

12 . The image encoding method of claim 11 , further comprising:
obtaining the bitrate increasing rate information by comparing an image compression result of the student neural network with the first initial search position applied and an image compression result of the student neural network without applying the first initial search position.

13 . The image encoding method of claim 9 , further comprising outputting a predicted frame by performing motion compensation based on result of the motion estimation.

14 . The image encoding method of claim 9 , wherein the loss function comprises a computation of a difference between the first initial search position and a ground truth label.

15 . An image decoding apparatus comprising:
an entropy decoder configured to receive a bitstream and parse the received bitstream to obtain information necessary for image reconstruction; an inverse quantizer configured to output a transform coefficient by inversely quantizing a quantized transform coefficient contained in the information; an inverse transformer configured to obtain a residual block by inversely transforming the transform coefficient; a predictor configured to generate a predicted block on the basis of information on prediction obtained from the entropy decoder; an adder configured to generate a reconstructed block by adding the residual block and the predicted block; and a filter configured to apply filtering to the reconstructed block, wherein the bitstream is an encoded image signal based on a motion vector obtained using an initial search position output by inputting a current frame and a reference frame into a student neural network trained through a knowledge distillation technique using a teacher neural network, and wherein the student neural network is trained by inputting a first frame and a second frame into the teacher neural network to obtain a first initial search position, inputting the first frame and the second frame into the student neural network to obtain a second initial search position, and training the student neural network based on a loss function using a difference between the first initial search position and the second initial search position.

16 . An image decoding method comprising:
receiving a bitstream and parsing the received bitstream to obtain information necessary for image reconstruction; outputting a transform coefficient by inversely quantizing a quantized transform coefficient contained in the information; obtaining a residual block by inversely transforming the transform coefficient; generating a predicted block on the basis of information on prediction obtained from an entropy decoder; generating a reconstructed block by adding the residual block and the predicted block; and applying filtering to the reconstructed block, wherein the bitstream is an encoded image signal based on a motion vector obtained using an initial search position output by inputting a current frame and a reference frame into a student neural network trained through a knowledge distillation technique using a teacher neural network, and wherein the student neural network is trained by inputting a first frame and a second frame into the teacher neural network to obtain a first initial search position, inputting the first frame and the second frame into the student neur

Assignees

• Samsung Electronics Co Ltd

Inventors

• Oh Paul
• Jee Seung Hoon
• Sung Young Hun
• Yang Jung Yeop
• Oh Do Kwan
• Lee Jun Hee
• Jang Dong Won
• Choi Jong Seong
• Hwang Chan Sol

Classifications

• H04N19/124

  Quantisation · CPC title

• H04N19/18

  the unit being a set of transform coefficients · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/91

  Entropy coding, e.g. variable length coding [VLC] or arithmetic coding · CPC title

• H04N19/117

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title