Federated teacher-student machine learning

Patent metadata

Field	Value
Publication number	US-2022012637-A1
Application number	US-202117370462-A
Country	US
Kind code	A1
Filing date	Jul 8, 2021
Priority date	Jul 9, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A node for a federated machine learning system that comprises the node and one or more other nodes configured for the same machine learning task, the node comprising:a federated student machine learning network configured to update a machine learning model in dependence upon updated machine learning models of the one or more node;a teacher machine learning network;means for receiving unlabeled data;means for teaching, using supervised learning, at least the federated first machine learning network using the teacher machine learning network, wherein the teacher machine learning network is configured to receive the data and produce pseudo labels for supervised learning using the data and wherein the federated student machine learning network is configured to perform supervised learning in dependence upon the same received data and the pseudo-labels.

First claim

1 . An apparatus for a federated machine learning system that comprises:
at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform;
update a machine learning model, with a federated student machine learning network, in dependence upon updated machine learning models of one or more other nodes;
wherein the apparatus is configured for a same machine learning task than the one or more other nodes;
teach, with a teacher machine learning network, by supervised learning, the federated student machine learning network,
wherein the teacher machine learning network is configured to produce pseudo-labels for the supervised learning by using received unlabeled data, and
wherein the federated student machine learning network is configured to perform supervised learning in dependence upon the received unlabeled data and the produced pseudo-labels.

2 . An apparatus as claimed in claim 1 , further comprising an adversarial machine learning network that is configured to cause to:
receive the unlabeled data, receive the produced pseudo-labels from the teacher machine learning network, receive label-estimates from the federated student machine learning network, and provide an adversarial loss to the teacher machine learning network, for training the teacher machine learning network.

3 . An apparatus as claimed in claim 1 , further comprising an adversarial machine learning network that is configured to cause to:
receive the unlabeled data, receive the produced pseudo-labels from the teacher machine learning network, receive label-estimates from the federated student machine learning network, and provide an adversarial loss to the federated student machine learning network for training the federated student machine learning network.

4 . An apparatus as claimed in claim 1 , further comprising an adversarial machine learning network that is configured to cause to:
receive the unlabeled data, receive the produced pseudo-labels from the teacher machine learning network, receive label-estimates from the federated student machine learning network, and provide an adversarial loss to the teacher machine learning network and the federated student machine learning network for training substantially simultaneously and/or parallelly the federated student machine learning network and the teacher machine learning network.

5 . An apparatus as claimed in claim 1 ,
wherein the supervised learning in dependence upon the received unlabeled data and the produced pseudo-labels further comprises supervised learning of the federated student machine learning network and, as an auxiliary task, unsupervised learning of the teacher machine learning network.

6 . An apparatus as claimed in claim 1 , further configured to cause to cluster by unsupervised learning of the teacher machine learning network so that intra-cluster mean distance is minimized and inter-cluster mean distance is maximized.

7 . An apparatus as claimed in claim 1 , wherein the teacher machine learning network is further configured to cause to cluster the received unlabeled data and the produced pseudo-labels so that intra-cluster mean distance is minimized and inter-cluster mean distance is maximized.

8 . An apparatus as claimed in claim 1 , wherein the federated student machine learning network is configured to update a student machine learning model of the federated student machine learning network in dependence upon updated one or more same first machine learning models of the one or more other nodes.

9 . An apparatus as claimed in claim 1 , wherein model parameters of the federated student machine learning network are used to update model parameters of one or more another student machine learning networks.

10 . An apparatus as claimed in claim 1 , wherein the federated student machine learning network is a student network and the teacher machine learning network is a teacher network configured to teach the student network.

11 . An apparatus as claimed in claim 1 , wherein the apparatus is a central node for the federated machine learning system,
wherein the one or more other node(s) are edge node(s) for the federated machine learning system, and wherein the federated machine learning system has a centralized federated machine learning system.

12 . A method for a federated machine learning system, comprising:
in a node, updating a machine learning model, with a federated student machine learning network, in dependence upon updated machine learning models of one or more other nodes; wherein the node is configured for a same machine learning task than the one or more other nodes; teaching, with a teacher machine learning network, by using supervised learning, the federated student machine learning network, wherein the teacher machine learning network is configured to produce pseudo-labels for the supervised learning by using received unlabeled data, and wherein the federated student machine learning network is configured to perform supervised learning in dependence upon the received unlabeled data and the produced pseudo-labels.

13 . A method as claimed in claim 12 , further comprising an adversarial machine learning network that is configured for:
receiving the unlabeled data, receiving the produced pseudo-labels from the teacher machine learning network, receiving label-estimates from the federated student machine learning network, and providing an adversarial loss to the teacher machine learning network for training the teacher machine learning network.

14 . A method as claimed in claim 12 , further comprising an adversarial machine learning network that is configured for:
receiving the unlabeled data, receiving the produced pseudo-labels from the teacher machine learning network, receiving label-estimates from the federated student machine learning network, and providing an adversarial loss to the federated student machine learning network for training the federated student machine learning network.

15 . A method as claimed in claim 12 , further comprising an adversarial machine learning network that is configured for:
receiving the unlabeled data, receiving the produced pseudo-labels from the teacher machine learning network, receiving label-estimates from the federated student machine learning network, and providing an adversarial loss to the teacher machine learning network and the federated student machine learning network for training substantially simultaneously and/or parallelly the federated student machine learning network and the teacher machine learning network.

16 . A method as claimed in claim 12 ,
wherein the supervised learning in dependence upon the received unlabeled data and the produced pseudo-labels further comprises supervised learning of the federated student machine learning network and, as an auxiliary task, unsupervised learning of the teacher machine learning network.

17 . A method as claimed in claim 12 , further configured for clustering by unsupervised learning of the teacher machine learning network so that intra-cluster mean distance is minimized and inter-cluster mean distance is maximized.

18 . A method as claimed in 12 , wherein the teacher machine learning network is further configured for clustering the received unlabeled data and the produced pseudo-labels so that intra-cluster mean distance is minimized and inter-cluster mean distance is maximized.

Assignees

• Nokia Technologies Oy

Inventors

• Rezazadegan Tavakoli Hamed
• Cricri Francesco
• Aksu Emre Baris

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/088 Primary

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title