Word segmentation method and system for language text

Patent metadata

Field	Value
Publication number	US-10691890-B2
Application number	US-201816134393-A
Country	US
Kind code	B2
Filing date	Sep 18, 2018
Priority date	Apr 12, 2016
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

A word segmentation method and system for a language text, where in the method, a word segmentation is performed on the first language text in a first word segmentation manner to obtain a first word boundary set, the first word boundary set is divided into a trusted second word boundary set and an untrusted third word boundary set according to a confidence level threshold, a second language text is selected from the first language text according to the third word boundary set, and a word segmentation is performed on the second language text in a second word segmentation manner to obtain a fourth word boundary set. Word segmentation precision of the first language text can be flexibly adjusted by adjusting the confidence level threshold.

First claim

What is claimed is:

1. A word segmentation method for a language text implemented by a computer device, comprising:
obtaining a first language text to be processed and a confidence level threshold, wherein the confidence level threshold indicates a word segmentation precision, a word segmentation speed, or a word segmentation consistency of the first language text;
performing word segmentation on the first language text in a first word segmentation manner to obtain a first word boundary set;
dividing the first word boundary set into a trusted second word boundary set and an untrusted third word boundary set according to the confidence level threshold;
selecting a second language text from the first language text according to the untrusted third word boundary set, wherein the second language text comprises a word corresponding to each word boundary in the untrusted third word boundary set;
performing the word segmentation on the second language text in a second word segmentation manner to obtain a fourth word boundary set, wherein a word segmentation precision of the second word segmentation manner is higher than a word segmentation precision of the first word segmentation manner;
setting the trusted second word boundary set and the fourth word boundary set as a word segmentation result of the first language text; and
outputting the word segmentation result of the first language text to at least one of an information retrieval system, a machine translation system, or a question answering system.

2. The method of claim 1 , wherein dividing the first word boundary set into the trusted second word boundary set and the untrusted third word boundary set comprises:
selecting at least one word corresponding to a word boundary from a context of each word boundary in the first word boundary set;
extracting a feature of the at least one word corresponding to each word boundary;
determining, a confidence level of each word boundary in the context using a classifier obtained by training in advance according to the feature of the at least one word corresponding to each word boundary;
adding a word boundary in the first word boundary set having a confidence level that is greater than the confidence level threshold to the trusted second word boundary set; and
adding a word boundary in the first word boundary set having a confidence level that is less than or equal to the confidence level threshold to the untrusted third word boundary set.

3. The method of claim 2 , wherein determining the confidence level of each word boundary in the context comprises determining the confidence level of each word boundary in the context according to an equation

P
⁡

(

True
|

B
i

,
c

)

=

S
⁡

(

True
,

B
i

,
c

)

∑
t

⁢

S
⁡

(

t
,

B
i

,
c

)

,

wherein the P(True|B i ,c) represents a confidence level in the context c of an i th word boundary (B i ) in the first word boundary set, the S(t,B i ,c) represents a score of the B i in the context c, the

S
⁡

(

t
,

B
i

,
c

)

=

∑
j

⁢

β
j

⁢

f
j

⁡

(

t
,

B
i

,
c

)

,

the f j (t,B i ,c) represents a j th feature in the feature of the at least one word, the β j represents a parameter of the classifier, the t represents a class corresponding to the classifier, and the t∈{True,False}.

4. The method of claim 2 , wherein selecting the at least one word corresponding to each word boundary comprises selecting a word corresponding to each word boundary from the context of each word boundary, wherein a word previous to the word corresponds to each word boundary, and wherein a word next to the word corresponds to each word boundary.

5. The method of claim 2 , wherein a parameter of the classifier comprises a parameter obtained by training based on a target language text, and wherein the target language text comprises a language text obtained after the word segmentation is performed in the first word segmentation manner on a language text having a word boundary that is known.

6. A computer device for segmenting a language text, comprising:
a memory comprising instructions; and
a processor coupled to the memory, wherein the instructions cause the processor to be configured to:
obtain a first language text to be processed and a confidence level threshold, wherein the confidence level threshold indicates a word segmentation precision, a word segmentation speed, or a word segmentation consistency of the first language text;
perform word segmentation on the first language text in a first word segmentation manner to obtain a first word boundary set;
divide the first word boundary set into a trusted second word boundary set and an untrusted third word boundary set according to the confidence level threshold;
select a second language text from the first l

Assignees

• Huawei Tech Co Ltd

Inventors

• Chen Xiao
• Li Hang

Classifications

• G06F18/285

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F40/53 Primary

  Processing of non-Latin text (kana-to-kanji conversion G06F40/129; vowelisation G06F40/232) · CPC title