Training a data coding system comprising a feature extractor neural network

Patent metadata

Field	Value
Publication number	US-12170779-B2
Application number	US-202117917153-A
Country	US
Kind code	B2
Filing date	Mar 30, 2021
Priority date	Apr 9, 2020
Publication date	Dec 17, 2024
Grant date	Dec 17, 2024

Abstract

Example embodiments provide a system for training a data coding pipeline including a feature extractor neural network, an encoder neural network, and a decoder neural network configured to reconstruct input data based on encoded features. A plurality of losses corresponding to different tasks may be determined for the coding pipeline. Tasks may be performed based on an output of the coding pipeline. A weight update may be determined for at least a subset of the coding pipeline based on the plurality of losses. The weight update may be configured to reduce a number of iterations for fine-tuning the coding pipeline for one of the tasks. This enables faster adaptation of the coding pipeline for one of the tasks after deployment of the coding pipeline. Apparatuses, methods, and computer programs are disclosed. Apparatuses, methods, and computer programs are disclosed.

First claim

The invention claimed is:

1. An apparatus, comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
obtain a data coding pipeline comprising a feature extractor neural network configured to extract features from input data, an encoder neural network configured to encode extracted features from the feature extractor neural network, and a decoder neural network configured to reconstruct the input data based on output of the encoder neural network;
determine a plurality of losses for the coding pipeline, the plurality of losses corresponding to at least a plurality of tasks;
determine a weight update for at least a subset of the coding pipeline based on the plurality of losses, wherein the weight update is configured to reduce a number of iterations for fine-tuning the coding pipeline for at least one of the plurality of tasks; and
update at least the subset of the coding pipeline based on the weight update.

2. The apparatus according to claim 1 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
determine the weight update based on an average performance of the plurality of tasks.

3. The apparatus according to claim 1 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
receive the input data for encoding;
extract features form the input data;
encode the extracted features with at least the encoder neural network to obtain an encoded bitstream; and
transmit the encoded bitstream to a decoder device.

4. The apparatus according to claim 1 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
fine-tune the feature extractor neural network based on the plurality of losses corresponding to at least the plurality of tasks.

5. The apparatus according to claim 4 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
fine-tune the encoder neural network based on the plurality of losses corresponding to at least the plurality of tasks.

6. The apparatus according to claim 4 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
fine-tune a plurality of task neural networks configured to perform the plurality of tasks based on the plurality of losses.

7. The apparatus according to claim 6 , wherein fine-tuning the plurality of task neural networks is based on a compression term configured to reduce entropy of a second plurality of weight updates associated with the plurality of task neural networks.

8. The apparatus according to claim 7 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
compress the second plurality of weight updates; and/or
transmit the second plurality of weight updates to a decoder device or at least one client encoder device.

9. The apparatus according to claim 4 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
determine the plurality of losses based on outputs of a plurality of task neural networks and ground-truth data for the plurality of tasks.

10. The apparatus according to claim 4 , wherein the at least one memory further stores instructions that, when executed by the at least one processor cause the apparatus at least to:
compute the plurality of losses based on outputs of a plurality of task neural networks and outputs of a plurality of reference neural networks configured to perform the plurality of tasks based on the input data.

11. The apparatus according to claim 1 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, a cause the apparatus to:
encode at least one portion of the input data; and
transmit the encoded at least one portion of the data to a decoder device.

12. An apparatus, comprising:
at least one processor; and
at least one memory including storing instructions that, when executed by the at least one processor, cause the apparatus to perform:
receive encoded features of input data;
receive at least one encoded portion of the input data;
decode the at least one encoded portion of the input data to form a decoded portion of the input data;
extract features from the decoded portion of the input data with a feature extractor neural network;
determine, with a decoder neural network, decoded features based on the encoded features;
perform a plurality of tasks with a plurality of task neural networks based on the decoded features;
perform the plurality of tasks with a plurality of reference task neural networks based on the extracted features, wherein the plurality of task neural networks are independent of the plurality of reference task neural networks;
determine a plurality of losses based on outputs of the plurality of task neural networks and outputs of the plurality of reference task neural networks;
determine a weight update based on the plurality of losses; and
fine-tune the feature extractor neural network based on the weight update, wherein the weight update is configured to reduce a number of iterations for the fine-tuning.

13. The apparatus according to claim 12 , wherein the at least one memory further stores instructions that, when executed by the at least one processor, cause the apparatus to:
compress the weight update; and/or
transmit the weight update to an encoder device.

14. A method, comprising:
obtaining a data coding pipeline comprising a feature extractor neural network configured to extract features from input data, an encoder neural network configured to encode extracted features from the feature extractor neural network, and a decoder neural network configured to reconstruct the input data based on output of the encoder neural network;
determining a plurality of losses for the coding pipeline, the plurality of losses corresponding to at least a plurality of tasks;
determining a weight update for at least a subset of the coding pipeline based on the plurality of losses, wherein the weight update is configured to reduce a number of iterations for fine-tuning the coding pipeline for at least one of the plurality of tasks; and
updating at least the subset of the coding pipeline based on the weight update.

15. The method according to claim 14 , further comprising:
fine-tuning the feature extractor neural network based on the plurality of losses corresponding to at least the plurality of tasks.

16. The method according to claim 15 , further comprising:
determining the plurality of losses based on outputs of a plurality of task neural networks and ground-truth data for the plurality of tasks.

17. The method according to claim 15 , further comprising:
computing the plurality of losses based on outputs of a plurality of task neural networks and outputs of a plurality of reference neural networks configured to perform the plurality of tasks based on the input data.

18. A method, comprising:
receiving encoded features of input data;
receiving at least one encoded portion of the input data;
decoding the at least one encoded portion of the input data to form a decoded portion of the input data;
extr

Assignees

• Nokia Technologies Oy

Inventors

• Cricri Francesco
• Le Nam
• Rezazadegan Tavakoli Hamed
• Zhang Honglei
• Hannuksela Miska Matias
• Aksu Emre Baris

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title