Intent classification using non-correlated features

Patent metadata

Field	Value
Publication number	US-11966699-B2
Application number	US-202117350116-A
Country	US
Kind code	B2
Filing date	Jun 17, 2021
Priority date	Jun 17, 2021
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

A system for classifying a language sample intent by receiving a language sample including a set of features, identifying language sample features, determining a tokenization score for the language sample according to the language sample features, eliminating duplicate features according to the tokenization score, determining a term frequency (tf) according to the identified features and the tokenization score, determining an inverse document frequency (idf) according to the identified features and the tokenization score, and generating a term frequency-inverse document frequency (tf-idf) matrix for the identified features.

First claim

What is claimed is:

1. A computer system comprising computer processors to implement a method for classifying a language sample, the computer system comprising:
one or more computer processors;
one or more computer readable storage devices;
stored program instructions on one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising:
program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample,
determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
program instructions to receive a first language sample comprising a set of features;
program instructions to identify language sample features of the first language sample;
program instructions to determine a tokenization score for the first language sample according to the language sample features;
program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
program instructions to determine an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
program instructions to generate a revised term frequency-inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf, and the idf;
program instructions to receive input text;
program instructions to identify tokens in the input text; and
program instructions to translate the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.

2. The computer system according to claim 1 , the stored program instructions further comprising program instructions to identify the language according to language characteristics, white-space proportions, and average token length; and
program instructions to determine the tokenization score according to the language and the language features.

3. The computer system according to claim 1 , wherein determining a tokenization score for the language sample according to the language comprises determining a tokenization score according to information including a white-space proportion estimate and an average token length.

4. The computer system according to claim 1 , the stored program instructions further comprising program instructions to eliminate duplicate features according to the tokenization score.

5. The computer system according to claim 4 , wherein eliminating duplicate features comprises keeping white-space features for a high tokenization score.

6. The computer system according to claim 4 , wherein eliminating duplicate features comprises keeping sentencepiece features for a low tokenization score.

7. The computer system according to claim 1 , the stored program instructions further comprising program instructions to train a language sample classifier according to training sample features and tokenization scores.

8. A method for classifying a language sample, the method comprising:
training, by one or more computer processors, a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample, determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
receiving, by the one or more computer processors, a first language sample comprising a set of features;
identifying, by the one or more computer processors, language sample features of the first language sample;
determining, by the one or more computer processors, a tokenization score for the first language sample according to the language sample features;
determining, by the one or more computer processors, a term frequency (tf) according to the identified language sample features and the tokenization score;
determining, by the one or more computer processors, an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
generating, by the one or more computer processors, a revised term frequency—inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf and the idf;
receiving, by the one or more computer processors, input text;
identifying, by the one or more computer processors, tokens in the input text; and
translating, by the one or more computer processors, the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.

9. The method according to claim 8 , further comprising identifying, by the one or more computer processors, the language according to language characteristics, white-space proportions, and average token length; and
determining, by the one or more computer processors, the tokenization score according to the language and the language features.

10. The method according to claim 8 , wherein determining a tokenization score for the language sample according to the language comprises determining a tokenization score according to information including a white-space proportion estimate and an average token length.

11. The method according to claim 8 , further comprising eliminating, by the one or more computer processors, duplicate features according to the tokenization score.

12. The method according to claim 11 , wherein eliminating duplicate features comprises keeping white-space features for a high tokenization score.

13. The method according to claim 11 , wherein eliminating duplicate features comprises keeping sentencepiece features for a low tokenization score.

14. The method according to claim 8 , further comprising training, by the one or more computer processors, a language sample classifier according to training sample features and tokenization scores.

15. A computer program product for classifying a language sample, the computer program product comprising one or more computer readable storage devices and collectively stored program instructions on the one or more computer readable storage devices, the stored program instructions comprising:
program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample,
determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
program instructions to receive a first language sample comprising a set of features; program instructions to identify language sample features of the first language sample;
program instructions to determine a tokenization score for the first language sample according to the language sample features;
program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
program instructions to determine an inverse document frequency (idf) according to the identi

Assignees

• IBM

Inventors

• Shah Abhishek
• Kunc Ladislav
• Qi Haode
• Pan Lin
• Potdar Saloni

Classifications

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F16/3344

  using natural language analysis · CPC title

• G06F16/355

  Creation or modification of classes or clusters · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N20/00

  Machine learning · CPC title