Sentence embedding method and apparatus based on subword embedding and skip-thoughts

Patent metadata

Field	Value
Publication number	US-11423238-B2
Application number	US-201916671773-A
Country	US
Kind code	B2
Filing date	Nov 1, 2019
Priority date	Dec 4, 2018
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

Provided are sentence embedding method and apparatus based on subword embedding and skip-thoughts. To integrate skip-thought sentence embedding learning methodology with a subword embedding technique, a skip-thought sentence embedding learning method based on subword embedding and methodology for simultaneously learning subword embedding learning and skip-thought sentence embedding learning, that is, multitask learning methodology, are provided as methodology for applying intra-sentence contextual information to subword embedding in the case of subword embedding learning. This makes it possible to apply a sentence embedding approach to agglutinative languages such as Korean in a bag-of-words form. Also, skip-thought sentence embedding learning methodology is integrated with a subword embedding technique such that intra-sentence contextual information can be used in the case of subword embedding learning. A proposed model minimizes additional training parameters based on sentence embedding such that most training results may be accumulated in a subword embedding parameter.

First claim

What is claimed is:

1. A sentence embedding method for outputting a sentence embedding result from an input sentence on the basis of subword embedding and skip-thoughts, the method comprising:
separating words of the input sentence for token separation;
extracting subwords from the words determined in the separating of words;
deriving subword embedding vector values by embedding the extracted subwords when the extracting of subwords is finished;
determining position encoding values by performing syntagma-based position encoding using fixed weights according to word positions in the input sentence after the deriving of subword embedding vector values in order for sentence embedding calculation; and
performing the sentence embedding calculation based on the subword embedding vector values and the position encoding values,
wherein the deriving subword embedding vector values comprises:
generating a subword table including a {word: subword set} dictionary and a {subword: vector value} table by separating the words and extracting the subwords from training text including consecutive sentence context;
generating subword embedding training data of {target word, contextual word} for subword embedding learning;
generating skip-thought sentence embedding training data of {target sentence, contextual sentence} for skip-thought sentence embedding learning; and
constructing a subword embedding and skip-thought sentence embedding integration model from the subword embedding training data and the skip-thought sentence embedding training data and generating the subword embedding vector values which are final training results, and

wherein the performing the sentence embedding calculation comprises multiplying the subword embedding vector values and the position encoding values together to output products and calculating an average of the products regarding the whole input sentence.

2. The sentence embedding method of claim 1 , wherein the subword embedding and skip-thought sentence embedding integration model finds a subword embedding value Φ t which maximizes a log likelihood function L as shown in the following equation:

ℒ
=

∑

t
=
1

T
w

⁢

⁢

∑

c
∈

C
t

⁢

log
⁢

⁢

p
⁡

(

w
c

❘

w
t

)

+

∑

t
=
1

T
s

⁢

⁢

∑

n
∈

N
t

⁢

log
⁢

⁢

p
⁡

(

sent
n

❘

sent
t

)

(where T w denotes a size of subword embedding training data, T s denotes a size of skip-thought sentence embedding training data, C t denotes a set of contextual words w c of w t , and N t denotes a contextual sentence set of a target sentence sent t ).

3. The sentence embedding method of claim 1 , wherein the syntagma-based position encoding comprises applying differentiated weights to full morphemes and bound morphemes constituting the input sentence.

4. A subword embedding and skip-thought sentence embedding integration model construction method for generating the subword embedding vector values required for performing the sentence embedding method of claim 1 , wherein the subword embedding and skip-thought sentence embedding integration model comprises:
the subword table including a {word: subword set} dictionary and a {subword: vector value} table;
the subword embedding training data of {target word, contextual word}; and
the skip-thought sentence embedding training data of {target sentence, contextual sentence}.

5. The subword embedding and skip-thought sentence embedding integration model construction method of claim 4 , wherein the subword embedding and skip-thought sentence embedding integration model finds a subword embedding value Φ t which maximizes a log likelihood function L as shown in the following equation:

ℒ
=

∑

t
=
1

T
w

⁢

⁢

∑

c
∈

C
t

⁢

log
⁢

⁢

p
⁡

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Chung Eui Sok
• Kim Hyun Woo
• Song Hwa Jeon
• Jung Ho Young
• Kang Byung Ok
• Park Jeon Gue
• Oh Yoo Rhee
• Lee Yun Keun

Classifications

• G06F40/289

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/216

  using statistical methods · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F40/56 Primary

  Natural language generation · CPC title