Input-feeding architecture for attention based end-to-end speech recognition

Patent metadata

Field	Value
Publication number	US-10672382-B2
Application number	US-201816160352-A
Country	US
Kind code	B2
Filing date	Oct 15, 2018
Priority date	Oct 15, 2018
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

Methods and apparatuses are provided for performing end-to-end speech recognition training performed by at least one processor. The method includes receiving, by the at least one processor, one or more input speech frames, generating, by the at least one processor, a sequence of encoder hidden states by transforming the input speech frames, computing, by the at least one processor, attention weights based on each of the sequence of encoder hidden states and a current decoder hidden state, performing, by the at least one processor, a decoding operation based on a previous embedded label prediction information and a previous attentional hidden state information generated based on the attention weights; and generating a current embedded label prediction information based on a result of the decoding operation and the attention weights.

First claim

What is claimed is:

1. A method of performing end-to-end speech recognition training performed by at least one processor, the method comprising:
receiving, by the at least one processor, one or more input speech frames;
generating, by the at least one processor, a sequence of encoder hidden states by transforming the input speech frames;
generating, by the at least one processor, a current hidden state of a decoder by performing a decoding operation based on a previous embedded label prediction information and a previous attentional hidden state information;
generating, by the at least one processor, context vectors by computing attention weights based on each of the sequence of encoder hidden states and the current hidden state of the decoder;
generating, by the at least one processor, a current attentional hidden state based on the context vector and the current hidden state of the decoder; and
generating, by the at least one processor, an output sequence based on the attentional hidden state.

2. The method of claim 1 , wherein the output sequence is a current label prediction information generated by performing a projection and softmax operation based on the attentional hidden state.

3. The method of claim 1 , wherein the computing the attention weights further comprises calculating a compatibility score, which represents alignment between the current decoder hidden state and each of the encoded hidden states.

4. The method of claim 3 , wherein the computing the attention weights is calculated by the following equation:

a

t
,
i

=

align
⁢

⁢

(

h
i
dec

,

h
enc

)

=

exp
⁡

(

score
⁡

(

h
i
dec

,

h
t
enc

)

)

∑

t
′

=
1

T

⁢

exp
⁡

(

score
⁡

(

h
i
dec

,

h

t
′

enc

)

)

.

5. The method of claim 4 ,
wherein the score(h i dec ,h t enc )=v a T ∘ tanh(W a [h i dec ;h t enc ]),
where [a;b] denotes concatenation of two vectors.

6. The method of claim 1 , wherein the context vector, c i is calculated based on a weighted sum of the sequence of encoder hidden states as follows:

c
i

=

∑

t
=
1

T

⁢

a

t
,
i

⁢

h
t
enc

.

7. The method of claim 1 , wherein the current attentional hidden state h′ i dec is obtained as follows: h′ i dec =tanh(W h [c i ;h i dec ]).

8. An end-to-end speech recognition training apparatus comprising:
at least one memory operable to store program code; and
at least one processor operable to read said program code and operate as instructed by said program code, said program code comprising:
receive one or more input speech frames;
generate a sequence of encoder hidden states by transforming the input speech frames;
generate a current hidden state of a decoder by performing a decoding operation based on a previous embedded label prediction information and a previous attentional hidden state information;
generate context vectors by computing attention weights based on each of the sequence of encoder hidden states and the current hidden state of the decoder;
generate a current attentional hidden state based on the context vector and the current hidden state of the decoder; and
generate an output sequence based on the attentional hidden state.

9. The end-to-end speech recognition training apparatus of claim 8 , wherein the output sequence is a current label prediction information generated by performing a projection and softmax operation based on the attentional hidden state.

10. The end-to-end speech recognition training apparatus of claim 8 , wherein the computing the attention weights further comprises calculating a compatibility score, which represents alignment between the current decoder hidden state and each of the encoded hidden states.

11. The end-to-end speech recognition training apparatus of claim 10 , wherein the computing the attention weights is ca

Assignees

• Tencent America LLC

Inventors

• Weng Chao
• Cui Jia
• Wang Guangsen
• Wang Jun
• Yu Chengzhu
• Su Dan
• Yu Dong

Classifications

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L15/16

  using artificial neural networks · CPC title