Deep video coding with block-based motion estimation

Patent metadata

Field	Value
Publication number	US-2025386027-A1
Application number	US-202519302635-A
Country	US
Kind code	A1
Filing date	Aug 18, 2025
Priority date	Feb 22, 2023
Publication date	Dec 18, 2025
Grant date	—

Abstract

An apparatus for determining an encoding of a motion field for a picture of a video sequence comprising a sequence of pictures, such that said picture is decodable using a reference picture, the motion field and the residual, according to an embodiment is provided. The apparatus comprises a trained neural network configured to determine the encoding of the motion field, being associated with said picture, depending on said picture and depending on the reference picture.

First claim

1 . An apparatus for determining an encoding of a motion field for a picture of a video sequence comprising a sequence of pictures, such that said picture is decodable using a reference picture, the motion field and the residual,
wherein the apparatus comprises a trained neural network configured to determine the encoding of the motion field, being associated with said picture, depending on said picture and depending on the reference picture.

2 . An apparatus for encoding,
wherein the apparatus is configured to encode a video sequence comprising a sequence of pictures to acquire encoded video data, wherein the apparatus is configured to generate the encoded video data such that each picture of one or more pictures of the video sequence is encoded by an encoding of a motion field and a residual, such that said picture is decodable using a reference picture, the motion field and the residual; wherein the apparatus comprises a trained neural network configured to determine the encoding of the motion field, being associated with said picture, depending on said picture and depending on the reference picture.

3 . An apparatus according to claim 1 ,
wherein the apparatus is configured to determine the motion field using a block-based motion search strategy, and wherein the trained neural network is configured to determine the encoding of the motion field.

4 . An apparatus according to claim 1 ,
wherein the apparatus is configured to determine two or more motion fields using the block-based motion search strategy, wherein the trained neural network is configured to determine the encoding of the motion field depending on the two or more motion fields that have been determined using the block-based motion search strategy.

5 . An apparatus according to claim 4 ,
wherein the apparatus is configured to determine the encoding of the motion field depending on the two or more motion fields by employing a cost function.

6 . An apparatus according to claim 4 ,
wherein the two or more motion fields exhibit different block sizes, for example, 8×8, and/or 16×16, and/or 32×32, and/or 64×64.

7 . An apparatus according to claim 3 ,
wherein the apparatus is configured to determine the motion field or the one or more motion fields using the block-based motion search strategy without using a neural network, and wherein the trained neural network is configured to determine the encoding of the motion field depending on the motion field or depending on the one or more motion fields.

8 . An apparatus according to claim 3 ,
wherein the block-based motion strategy comprises a block-based diamond search.

9 . An apparatus according to claim 3 ,
wherein the block-based motion strategy comprises a to determine the motion field depending on a sub-pel search.

10 . An apparatus according to claim 1 ,
wherein the trained neural network has been trained using a minimization function or optimization function, which depends on a predicted picture and an original picture, wherein the predicted picture is a picture that results from decoding using a reference picture and a motion field which are associated with said predicted picture.

11 . An apparatus according to claim 10 ,
wherein the neural network has been trained comprising minimizing a mean squared error between a predicted picture and an original picture.

12 . An apparatus according to claim 10 ,
wherein the neural network has been trained comprising minimizing a rate which depends on the motion field and/or on a residual.

13 . An apparatus according to claim 12 ,
wherein the neural network has been trained comprising the minimizing of a rate which depends on a rate of a block-based transform coder for the residual.

14 . An apparatus according to claim 12 ,
wherein the neural network has been trained comprising the minimizing of the rate depending on

∑
k

-

log
2

⁢

P
z

(

z
~

k

,

(

μ
k

^

,

σ
k

^

)

)

+

∑
l

-

log
2

⁢

P
y

(

y
~

l

,
ϕ

)

,

and/or depending on

κ
⁢

∑

𝔅
j



DCT
(

x

i
+
1

Assignees

• Fraunhofer Ges Forschung

Inventors

• Pientka Sophie
• Schäfer Michael
• Pfaff Jonathan
• Schwarz Heiko
• Marpe Detlev
• Wiegand Thomas

Classifications

• H04N19/42

  characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) · CPC title

• H04N19/176 Primary

  the region being a block, e.g. a macroblock · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title