Language recognition based on vocabulary lists

Patent metadata

Field	Value
Publication number	US-9336197-B2
Application number	US-201314108224-A
Country	US
Kind code	B2
Filing date	Dec 16, 2013
Priority date	Jan 22, 2013
Publication date	May 10, 2016
Grant date	May 10, 2016

Abstract

A method is implemented at a computer to determine that certain information content is composed or compiled in a specific language selected among two or more similar languages. The computer integrates a first vocabulary list of a first language and a second vocabulary list of a second language into a comprehensive vocabulary list. The integrating includes analyzing the first vocabulary list in view of the second vocabulary list to identify a first vocabulary sub-list that is used in the first language, but not in the second language. The computer then identifies, in the information content, a plurality of expressions that are included in the comprehensive vocabulary list, and a subset of expressions that are included in the first vocabulary sub-list. Upon a determination that a total frequency of occurrence of the subset of expressions meets predetermined occurrence criteria, the computer determines that the information content is composed in the first language.

First claim

What is claimed is:

1. A computer-implemented method of recognizing a first language used in information content, comprising:
at a computer having one or more processors and memory for storing programs to be executed by the one or more processors:
integrating a first vocabulary list and a second vocabulary list that are built based on a first language and a second language, respectively, into a comprehensive vocabulary list, wherein the integrating includes analyzing the first vocabulary list in view of the second vocabulary list to at least identify a first vocabulary sub-list, in the comprehensive vocabulary list, that is used in the first language, but not in the second language;
identifying, within the information content, a plurality of expressions that are included in the comprehensive vocabulary list;
identifying, within the plurality of expressions, a subset of expressions that are included in the first vocabulary sub-list;
determining that a total frequency of occurrence of the subset of expressions within the information content meets predetermined occurrence criteria; and
in accordance with the determination, determining that the information content is composed in the first language.

2. The method of claim 1 , wherein building the first vocabulary list and the second vocabulary list, further comprising:
creating the respective vocabulary list for either one of the first language and the second language;
identifying a plurality of test expressions in test information content corresponding to the respective language;
after identifying, computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions; and
when the respective probability value for each test expression exceeds a respective test threshold value, modifying the respective vocabulary list by incorporating the respective test expression.

3. The method of claim 2 , wherein computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions, further comprising:
determining a correlation value, a plurality of left entropy values and a plurality of right entropy values for the respective test expression in the plurality of test words, the correlation value being related to mutual information dependence among words in the respective test expression, each of the plurality of right entropy values being associated with a probability of coupling a respective right expression, in the respective language, to the right of the respective test expression, each of the plurality of left entropy values being associated with a probability of coupling a respective left expression, in the respective language, to the left of the respective test expression; and
computing the respective probability value for the respective test expression by summing up the correlation value, the plurality of left entropy values and the plurality of right entropy values in a weighted manner.

4. The method of claim 1 , wherein the subset of expressions are included in the first vocabulary sub-list, when each word in the subset of expressions is included in the first vocabulary list, but not in the second vocabulary list.

5. The method of claim 1 , wherein determining that, in the information content, the total frequency of occurrence for the subset of expressions meets the predetermined occurrence criteria, based on at least one parameter among the number of words in the subset of expressions, a first ratio between the number of words in the subset of expressions and the total number of words in the information content, and a second ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary.

6. The method of claim 1 , wherein the total frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when the number of words in the subset of expressions exceeds a threshold number.

7. The method of claim 1 , wherein the total frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when a ratio between the number of words in the subset of expressions and the total number of words in the information content exceed a first threshold ratio.

8. The method of claim 1 , wherein the frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when a ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary exceeds a second threshold ratio.

9. The method of claim 1 , wherein determining that, in the information content, the total frequency of occurrence for the subset of expressions meets the predetermined occurrence criteria, further comprising:
computing the number of words in the subset of expressions that are specifically associated with the first language;
computing a first ratio between the number of words in the subset of expressions and the total number of words in the information content;
computing a second ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary;
calculating a weighted sum of the number of words in the subset of expressions, the first ratio and the second ratio; and
determining that the weighted sum exceeds a predetermined threshold value.

10. The method of claim 1 , wherein the first and second languages are two distinct Chinese dialects, and wherein identifying the plurality of expressions, within the information content, that are included in the comprehensive vocabulary list, further comprises:
segmenting the information content to the plurality of expressions in accordance with the comprehensive vocabulary list.

11. The method of claim 1 , wherein the first and second languages are two distinct languages selected from a language group that consists of British English, American English and Australian English.

12. The method of claim 1 , wherein the second vocabulary list is analyzed in view of the first vocabulary list to identify a second vocabulary sub-list, in the comprehensive vocabulary list, that is used in the second language, but not in the first language, the subset of expressions in the plurality of expressions comprising a first subset of expressions, the total frequency of occurrence comprising a first frequency of occurrence, the method further comprising:
identifying a second subset of expressions, in the plurality of expressions, that are specifically associated with the second language and included in the second vocabulary sub-list;
determining, in the information content, a second frequency of occurrence for the second subset of expressions; and
determining a primary language used by the information content according to a comparison between the first frequency of occurrence and the second frequency of occurrence.

13. The method of claim 1 , wherein the first vocabulary list and the second vocabulary list are analyzed to identify a mixed vocabulary sub-list and a second vocabulary sub-list in the comprehensive vocabulary list, the mixed vocabulary sub-list being used in both the first and second languages, the second vocabulary sub-list being used in the second language, but not in the first language, the total frequency of occurrence comprising a first frequency of occurrence, the method further comprising

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Li Lu
• Cheng Qiang
• Ma Jianxiong
• Rao Feng
• Lu Duling
• Lu Li
• Zhang Xiang
• Chen Bo

Classifications

• G06F40/242 Primary

  Dictionaries · CPC title

• G06F17/2735 Primary

  Physics · mapped topic