Parameter updating method and apparatus and storage medium

Patent metadata

Field	Value
Publication number	US-12511518-B2
Application number	US-202217993436-A
Country	US
Kind code	B2
Filing date	Nov 23, 2022
Priority date	May 26, 2020
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

The present disclosure relates to parameter updating methods. In one example method, a parameter in a neural network model is updated for a plurality of times through a plurality of iterations. The plurality of iterations include a first iteration period and a second iteration period. In the first iteration period, an inverse matrix of an additional matrix of the neural network model is updated once based on a quantity of iterations indicated by a first update stride. In the second iteration period, the inverse matrix of the additional matrix of the neural network model is updated once based on a quantity of iterations indicated by a second update stride, where the first iteration of the second iteration period is after the last iteration of the first iteration period in an iteration sequence, and the second update stride is greater than the first update stride.

First claim

What is claimed is:

1 . A parameter updating method, wherein the parameter updating method updates a parameter in a neural network model for a plurality of times through a plurality of iterations, the plurality of iterations comprise a first iteration period and a second iteration period, and the method comprises:
updating, in the first iteration period, an inverse matrix of an additional matrix of the neural network model once based on a quantity of iterations indicated by a first update stride, wherein the first iteration period comprises at least two iterations; and updating, in the second iteration period, the inverse matrix of the additional matrix of the neural network model once based on a quantity of iterations indicated by a second update stride, wherein the second iteration period comprises at least two iterations, a first iteration of the second iteration period is after a last iteration of the first iteration period in an iteration sequence, and the second update stride is greater than the first update stride.

2 . The method according to claim 1 , wherein the plurality of iterations comprise a third iteration period, the third iteration period is any iteration period of the plurality of iterations, and the method further comprises:
when an N th iteration in the plurality of iterations is in the third iteration period, and is an iteration in which an inverse matrix needs to be updated and that is indicated by a third update stride, updating the inverse matrix of the additional matrix of the neural network model, and updating the parameter in the neural network model by using an updated inverse matrix of the additional matrix and a first-order gradient of the N th iteration, wherein the third update stride is an update stride of the third iteration period, N is an integer, and N>1.

3 . The method according to claim 2 , wherein the updating the inverse matrix of the additional matrix of the neural network model, and updating the parameter in the neural network model by using an updated inverse matrix of the additional matrix and a first-order gradient of the N th iteration comprises:
updating inverse matrices of additional matrices of P blocks, wherein the P blocks are some or all of Q blocks of the neural network model, P and Q are integers, Q≥P, Q≥2, and P≥1; updating a parameter of a corresponding block in the P blocks by using updated inverse matrices of the additional matrices of the P blocks and a first-order gradient of the P blocks in the N th iteration; and when Q>P, for (Q−P) blocks other than the P blocks, updating a parameter of a corresponding block in the (Q−P) blocks by using inverse matrices of additional matrices used by the (Q−P) blocks in an (N−1) th iteration and a first-order gradient of the (Q−P) blocks in the N th iteration.

4 . The method according to claim 3 , wherein the method further comprises:
obtaining the P blocks from M blocks in the neural network model based on information about additional matrices of the M blocks, wherein the information about the additional matrix comprises a trace of the additional matrix or a 2-norm of the additional matrix, the M blocks are blocks that are in the Q blocks in the N th iteration and whose additional matrices need to be updated, M is an integer, and Q≥M≥P.

5 . The method according to claim 4 , wherein the obtaining the P blocks from M blocks in the neural network model based on information about additional matrices of the M blocks comprises:
obtaining the P blocks from the M blocks based on a trace of additional matrices of the M blocks in the N th iteration and a trace of additional matrices of the M blocks in the (N−1) th iteration.

6 . The method according to claim 5 , wherein the obtaining the P blocks from the M blocks based on a trace of additional matrices of the M blocks in the N th iteration and a trace of additional matrices of the M blocks in the (N−1) th iteration comprises:
obtaining the P blocks whose first ratio is greater than a first threshold from the M blocks, wherein the first ratio is a ratio of a first difference to the trace of the additional matrices in the (N−1) th iteration, and the first difference is a difference between the trace of the additional matrices in the N th iteration and the trace of the additional matrices in the (N−1) th iteration.

7 . The method according to claim 3 , wherein the method further comprises:
obtaining the P blocks from a plurality of blocks in the neural network model based on sampling probabilities of the plurality of blocks, wherein a sampling probability of a block indicates a probability that an inverse matrix of an additional matrix of the block is updated in the N th iteration.

8 . The method according to claim 2 , wherein the method further comprises:
updating the inverse matrix when a second difference in the N th iteration is equal to an update start value, wherein the second difference is a difference between N and a total length of a previous iteration period, the previous iteration period is located before the third iteration period in the iteration sequence, and the update start value indicates an iteration in which the inverse matrix is updated for the first time in the third iteration period.

9 . The method according to claim 2 , wherein the method further comprises:
updating the inverse matrix when a first remainder in the N th iteration is 0, wherein the first remainder is a remainder between a third difference and the third update stride, the third difference is a difference between an (N−an update start value) and a total length of a previous iteration period, the previous iteration period is located before the third iteration period in the iteration sequence, and the update start value indicates an iteration in which the inverse matrix is updated for the first time in the third iteration period.

10 . The method according to claim 1 , wherein in the first iteration period, the inverse matrix of the additional matrix of the neural network model is updated once every quantity of iterations indicated by the first update stride.

11 . A parameter updating method, wherein the parameter updating method updates a parameter in a neural network model for a plurality of times through a plurality of iterations, for an N th iteration in the plurality of iterations, N is an integer greater than 1, and the method comprises:
updating inverse matrices of additional matrices of P blocks, wherein the P blocks are some or all of Q blocks of the neural network model, P and Q are integers, Q≥P, Q≥2, and P≥1; updating a parameter of a corresponding block in the P blocks by using updated inverse matrices of the additional matrices of the P blocks and a first-order gradient of the P blocks in the N th iteration; and when Q>P, for (Q−P) blocks other than the P blocks, updating a parameter of a corresponding block in the (Q−P) blocks by using inverse matrices of additional matrices used by the (Q−P) blocks in an (N−1) th iteration and a first-order gradient of the (Q−P) blocks in the Nth iteration.

12 . The method according to claim 11 , wherein the method further comprises:
obtaining the P blocks from M blocks in the neural network model based on information about additional matrices of the M blocks, wherein the information about the additional matrix comprises a trace of the additional matrix or a 2-norm of the additional matrix, the M blocks are blocks that are in the Q blocks in the N th iteration and whose additional matrices need to be updated, M is an integer, and Q≥M≥P.

13 . The method according to claim 12 , wherein the obtaining the P blocks from M blocks in the ne

Assignees

• Huawei Tech Co Ltd

Inventors

• Wang Zidong
• Chen Mengyun
• Yu Fan
• Chen Lei

Classifications

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/08 Primary

  Learning methods · CPC title