Early exit for natural language processing models

Patent metadata

Field	Value
Publication number	US-2019266236-A1
Application number	US-201916411763-A
Country	US
Kind code	A1
Filing date	May 14, 2019
Priority date	May 14, 2019
Publication date	Aug 29, 2019
Grant date	—

Abstract

The disclosure provides a natural language processing (NLP) model arranged to operate on two lexicons, where one lexicon is a sub-set of the other lexicon. The NLP model can be arranged to generate output based on the sub-set lexicon and exit processing of the NLP model, to potentially save computation cycles.

First claim

What is claimed is:

1 . An apparatus, comprising:
a processor; and memory storing instructions and a natural language processing (NLP) inference model, the instructions when executed by the processor cause the processor to:
generate, via the NLP inference model, an intermediate result and a confidence associated with the intermediate result;
compare the confidence to a threshold; and
based on the comparison, either:
generate an output based on the intermediate result and cease computation via the NLP inference mode; or
generate the output via the NLP inference model.

2 . The apparatus of claim 1 , the memory storing instructions, which when executed by the processor cause the processor to:
determine whether the confidence is greater than or equal to the threshold; and generate the output based on the intermediate result and cease computation via the NLP inference mode based on a determination that the confidence is greater than or equal to the threshold.

3 . The apparatus of claim 2 , the memory storing instructions, which when executed by the processor cause the processor to generate the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

4 . The apparatus of claim 1 , the NLP model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, the memory storing instructions, which when executed by the processor cause the processor to derive the intermediate result and the confidence based on a first portion of the plurality of encoders and the first classifier.

5 . The apparatus of claim 4 , the memory storing instructions, which when executed by the processor cause the processor to derive the output based the plurality of the encoders and the second classifier.

6 . The apparatus of claim 4 , the memory storing instructions, which when executed by the processor cause the processor to not process a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutually exclusive of the first portion of the plurality of encoders.

7 . The apparatus of claim 4 , wherein the second lexicon comprises a vocabulary of a plurality of tokens and the first lexicon comprises a vocabulary including a sub-set of the plurality of tokens.

8 . The apparatus of claim 7 , wherein the vocabulary of the first lexicon is selected based in part on a statistical measurement of usage of the tokens of the vocabulary of the second lexicon.

9 . The apparatus of claim 1 , wherein the processor is an artificial intelligence (AI) accelerator.

10 . A non-transitory computer-readable storage medium, comprising instructions that when executed by a processor, cause the processor to:
generate, via an NLP inference model, an intermediate result and a confidence associated with the intermediate result; compare the confidence to a threshold; and based on the comparison, either:
generate an output based on the intermediate result and cease computation via the NLP inference mode; or
generate the output via the NLP inference model.

11 . The non-transitory computer-readable storage medium of claim 10 , comprising instructions that when executed by the processor, cause the processor to:
determine whether the confidence is greater than or equal to the threshold; and generate the output based on the intermediate result and cease computation via the NLP inference mode based on a determination that the confidence is greater than or equal to the threshold.

12 . The non-transitory computer-readable storage medium of claim 11 , comprising instructions that when executed by the processor, cause the processor to generate the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

13 . The non-transitory computer-readable storage medium of claim 10 , the NLP model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, the instructions when executed by the processor cause the processor to derive the intermediate result and the confidence based on a first portion of the plurality of encoders and the first classifier.

14 . The non-transitory computer-readable storage medium of claim 13 , comprising instructions that when executed by the processor, cause the processor to derive the output based the plurality of the encoders and the second classifier.

15 . The non-transitory computer-readable storage medium of claim 13 , comprising instructions that when executed by the processor, cause the processor to not process a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutually exclusive of the first portion of the plurality of encoders.

16 . The non-transitory computer-readable storage medium of claim 13 , wherein the second lexicon comprises a vocabulary of a plurality of tokens and the first lexicon comprises a vocabulary including a sub-set of the plurality of tokens and the vocabulary of the first lexicon is selected based in part on a statistical measurement of usage of the tokens of the vocabulary of the second lexicon.

17 . A computer-implemented method, comprising:
generating, via an NLP inference model, an intermediate result and a confidence associated with the intermediate result; comparing the confidence to a threshold; and based on the comparison, either:
generating an output based on the intermediate result and cease computation via the NLP inference mode; or
generating the output via the NLP inference model.

18 . The computer-implemented method of claim 17 , comprising:
determining whether the confidence is greater than or equal to the threshold; and generating the output based on the intermediate result and cease computation via the NLP inference mode based on a determination that the confidence is greater than or equal to the threshold.

19 . The computer-implemented method of claim 18 , comprising generating the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

20 . The computer-implemented method of claim 17 , the NLP model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, the method comprising:
deriving the intermediate result and the confidence based on a first portion of the plurality of encoders and the first classifier; and deriving the output based the plurality of the encoders and the second classifier.

21 . The computer-implemented method of claim 19 , comprising not processing a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutually exclusive of the first portion of the plurality of encoders.

22 . The computer-implemented method of claim 19 , wherein the second lexicon comprises a vocabulary of a plurality of tokens and the first lexicon comprises a vocabulary including a sub-set of the

Assignees

• Intel Corp

Inventors

• Battach Barak
• Bleiweiss Amit
• Barad Haim

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F40/216 Primary

  using statistical methods · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F17/2715

  Physics · mapped topic