Resource-efficient training of a sequence-tagging model

Patent metadata

Field	Value
Publication number	US-12511548-B2
Application number	US-202218075876-A
Country	US
Kind code	B2
Filing date	Dec 6, 2022
Priority date	Dec 6, 2022
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

A technique iteratively updates model weights of a teacher model and a student model. In operation, the teacher model produces noisy original pseudo-labeled training examples from unlabeled training examples. The technique weights the original pseudo-labeled training examples based on validation information. The technique then updates model weights of the student model based on the weighted pseudo-labeled training examples. The validation information, which is used to weight the original pseudo-labeled training examples, is produced by selecting labeled training examples based on an uncertainty-based factor and a similarity-based factor. The uncertainty-based factor describes an extent to which the student model produces uncertain classification results for the set of labeled training examples. The similarity-based factor describes the similarity between the set of labeled training examples and the unlabeled training examples. Overall, the technique is efficient because it eliminates the need to produce a large number labeled training examples.

First claim

What is claimed is:

1 . A computer-implemented method for generating a tagging model, comprising:
selecting a set of labeled training examples from a collection of labeled training examples, to form validation information, the collection of labeled training examples being less than a collection of unlabeled training examples, and the selecting including assessing appropriateness of a candidate labeled training example for inclusion in the validation information based on: an uncertainty measure that expresses an extent to which a student model produces an uncertain classification result for the candidate labeled training example; and a similarity measure that expresses an extent to which the candidate labeled training example is similar to unlabeled training examples in the collection of unlabeled training examples; producing original pseudo-labeled training examples using a teacher model based on the unlabeled training examples in the collection of unlabeled training examples; weighting the original pseudo-labeled training examples based on the validation information, to produce weighted pseudo-labeled training examples, the weighting reducing an effect of noise in the pseudo-labeled training examples; and updating model weights of the student model based on the weighted pseudo-labeled trained examples, the student model, once trained, corresponding to the tagging model.

2 . The computer-implemented method of claim 1 , further including repeating the selecting, producing, weighting, and updating plural times.

3 . The computer-implemented method of claim 2 , further including using the tagging model, following the repeating, to classify tokens in a sequence of tokens.

4 . The computer-implemented method of claim 3 , wherein the sequence of tokens describes characteristics of an identified product.

5 . The computer-implemented method of claim 1 , further comprising, after a prescribed number of repetitions of the selecting, producing, weighting, and updating, updating model weights of the teacher model based on current model weights of the student model.

6 . The computer-implemented method of claim 1 , further including, prior to the selecting, and for at least some of the collection of labeled training examples, masking at least part of the labeled training examples.

7 . The computer-implemented method of claim 6 , wherein the masking includes masking tokens that have been given an indeterminate classification status.

8 . The computer-implemented method of claim 1 , wherein the updating the model weights of the student model is also performed based on labeled training examples from the collection of labeled training examples.

9 . The computer-implemented method of claim 1 , further including generating the similarity measure by:
forming individual similarity measures, each individual similarity measure expressing similarity between the candidate labeled training example and a particular unlabeled training example; and forming an overall similarity measure based on the individual similarity measures.

10 . The computer-implemented method of claim 9 , wherein the overall similarity measure is an average of the individual similarity measures.

11 . A computing system, comprising:
a processing system comprising a processor; and a storage device for storing machine-readable instructions that, when executed by the processing system, perform operations comprising: classifying tokens in a sequence of tokens using a trained tagging model, to produce a classified sequence of tokens; and performing an application task based on the classified sequence of tokens, the trained tagging model being produced by training a student model in a training framework that includes the student model and a teacher model, model weights of the student model being updated based on pseudo-labeled training examples produced by the teacher model for unlabeled training examples in a collection of unlabeled training examples, the pseudo-labeled training examples being weighted based on validation information, and the validation information including a set of labeled training examples that are selected from a collection of labeled training examples based on plural factors, one of the plural factors being an assessed similarity between the set of labeled training examples and unlabeled training examples in the collection of unlabeled training examples.

12 . The computing system of claim 11 , wherein the sequence of tokens corresponds to a title of a product, wherein at least some of the tokens describe attributes of the product, and wherein the classifying classifies the types of the attributes.

13 . The computing system of claim 11 , wherein the performing an application task includes performing a search operation, matching operation, and/or filtering operation based on the classified sequence of tokens.

14 . The computing system of claim 11 , wherein another factor used to select the labeled set of training examples in the validation information is an assessed uncertainty of classification results produced by the student model for the labeled set of training examples.

15 . The computing system of claim 11 , wherein, for at least some of the set of labeled training examples, parts of the labeled training examples are masked.

16 . The computing system of claim 15 , wherein the masking includes masking tokens that have been given an indeterminate classification status.

17 . A computer-readable storage medium for storing computer-readable instructions, wherein a processing system executing the computer-readable instructions performs operations comprising:
selecting a set of labeled training examples from a collection of labeled training examples based on plural factors, to form validation information, one of the plural factors being an assessed similarity between the set of labeled training examples and unlabeled training examples in a collection of unlabeled training examples; producing original pseudo-labeled training examples using a teacher model based the unlabeled training examples in the collection of unlabeled training examples; weighting the original pseudo-labeled training examples based on the validation information, to produce weighted pseudo-labeled training examples; updating model weights of the student model based on the weighted pseudo-labeled trained examples; and repeating the selecting, producing, weighting, and updating plural times until a training objective is achieved, the student model, after the repeating, corresponding to a tagging model for use in classifying a sequence of tokens.

18 . The computer-readable storage medium of claim 17 , wherein another factor of the plural factors is an extent to which the student model produces uncertain classification results for the set of labeled training examples.

19 . The computer-readable storage medium of claim 17 , further including, prior to the selecting, masking at least part of labeled training examples in the collection of labeled training examples.

20 . The computer-readable storage medium of claim 19 , wherein the masking includes masking tokens that have been given an indeterminate classification status.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Cui Wen
• Chang Keng-Hao
• Lin Pai Chun
• Khalilishoja Mohammadreza
• Manavoglu Eren

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/096 Primary

  Transfer learning · CPC title