Early exit for natural language processing models

Patent metadata

Field	Value
Publication number	US-11544461-B2
Application number	US-201916411763-A
Country	US
Kind code	B2
Filing date	May 14, 2019
Priority date	May 14, 2019
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

The disclosure provides a natural language processing (NLP) model arranged to operate on two lexicons, where one lexicon is a sub-set of the other lexicon. The NLP model can be arranged to generate output based on the sub-set lexicon and exit processing of the NLP model, to potentially save computation cycles.

First claim

What is claimed is:

1. An apparatus, comprising:
a processor; and
memory storing instructions and a natural language processing (NLP) inference model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, the instructions when executed by the processor cause the processor to execute the NLP inference model on a single device and to:
generate, via the NLP inference model, an intermediate result and a confidence associated with the intermediate result based on a first portion of the plurality of encoders and the first classifier;
compare the confidence to a threshold; and
based on the comparison, either:
generate an output based on the intermediate result from the first portion of the plurality of encoders and the first classifier and cease computation via the NLP inference model; or
generate the output via the NLP inference model based on a second portion of the plurality of encoders and the second classifier, each encoder of the second portion of the plurality of encoders different from the first portion of the plurality of encoders.

2. The apparatus of claim 1 , the memory storing instructions, which when executed by the processor cause the processor to:
determine whether the confidence is greater than or equal to the threshold; and
generate the output based on the intermediate result and cease computation via the NLP inference model based on a determination that the confidence is greater than or equal to the threshold.

3. The apparatus of claim 2 , the memory storing instructions, which when executed by the processor cause the processor to generate the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

4. The apparatus of claim 1 , the memory storing instructions, which when executed by the processor cause the processor to derive the output based the plurality of the encoders and the second classifier.

5. The apparatus of claim 1 , the memory storing instructions, which when executed by the processor cause the processor to not process a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutually exclusive of the first portion of the plurality of encoders.

6. The apparatus of claim 1 , wherein the second lexicon comprises a vocabulary of a plurality of tokens and the first lexicon comprises a vocabulary including a sub-set of the plurality of tokens.

7. The apparatus of claim 6 , wherein the vocabulary of the first lexicon is selected based in part on a statistical measurement of usage of the tokens of the vocabulary of the second lexicon.

8. The apparatus of claim 1 , wherein the processor is an artificial intelligence (AI) accelerator.

9. A non-transitory computer-readable storage medium, comprising instructions that when executed by a processor, cause the processor to:
generate, via an NLP inference model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, an intermediate result and a confidence associated with the intermediate result based on a first portion of the plurality of encoders and the first classifier, wherein the NLP inference model is executed on a single device;
compare the confidence to a threshold; and
based on the comparison, either:
generate an output based on the intermediate result from the first portion of the plurality of encoders and the first classifier and cease computation via the NLP inference model; or
generate the output via the NLP inference model based on a second portion of the plurality of encoders and the second classifier, each encoder of the second portion of the plurality of encoders different from the first portion of the plurality of encoders.

10. The non-transitory computer-readable storage medium of claim 9 , comprising instructions that when executed by the processor, cause the processor to:
determine whether the confidence is greater than or equal to the threshold; and
generate the output based on the intermediate result and cease computation via the NLP inference model based on a determination that the confidence is greater than or equal to the threshold.

11. The non-transitory computer-readable storage medium of claim 10 , comprising instructions that when executed by the processor, cause the processor to generate the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

12. The non-transitory computer-readable storage medium of claim 9 , comprising instructions that when executed by the processor, cause the processor to derive the output based the plurality of the encoders and the second classifier.

13. The non-transitory computer-readable storage medium of claim 9 , comprising instructions that when executed by the processor, cause the processor to not process a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutually exclusive of the first portion of the plurality of encoders.

14. The non-transitory computer-readable storage medium of claim 9 , wherein the second lexicon comprises a vocabulary of a plurality of tokens and the first lexicon comprises a vocabulary including a sub-set of the plurality of tokens and the vocabulary of the first lexicon is selected based in part on a statistical measurement of usage of the tokens of the vocabulary of the second lexicon.

15. A computer-implemented method, comprising:
generating, via an NLP inference model comprising a plurality of encoders, a first classifier associated with a first lexicon and a second classifier associated with a second lexicon, where the first lexicon is a sub-set of the second lexicon, an intermediate result and a confidence associated with the intermediate result based on a first portion of the plurality of encoders and the first classifier, wherein the NLP inference model is executed on a single device;
comparing the confidence to a threshold; and
based on the comparison, either:
generating an output based on the intermediate result from the first portion of the plurality of encoders and the first classifier and cease computation via the NLP inference model; or
generating the output via the NLP inference model based on a second portion of the plurality of encoders and the second classifier, each encoder of the second portion of the plurality of encoders different from the first portion of the plurality of encoders.

16. The computer-implemented method of claim 15 , comprising:
determining whether the confidence is greater than or equal to the threshold; and
generating the output based on the intermediate result and cease computation via the NLP inference model based on a determination that the confidence is greater than or equal to the threshold.

17. The computer-implemented method of claim 16 , comprising generating the output via the NLP inference model based on a determination that the confidence is not greater than or equal to the threshold.

18. The computer-implemented method of claim 17 , comprising not processing a second portion of the plurality of encoders to cease computation via the NLP inference model, where the second portion of the plurality of encoders is mutu

Assignees

• Intel Corp

Inventors

• Battach Barak
• Bleiweiss Amit
• Barad Haim

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F40/216 Primary

  using statistical methods · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06N3/10

  Interfaces, programming languages or software development kits, e.g. for simulating neural networks · CPC title