Convolution-augmented transformer models

Patent metadata

Field	Value
Publication number	US-12079703-B2
Application number	US-202017139525-A
Country	US
Kind code	B2
Filing date	Dec 31, 2020
Priority date	Dec 31, 2020
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Systems and methods can utilize a conformer model to process a data set for various data processing tasks, including, but not limited to, speech recognition, sound separation, protein synthesis determination, video or other image set analysis, and natural language processing. The conformer model can use feed-forward blocks, a self-attention block, and a convolution block to process data to learn global interactions and relative-offset-based local correlations of the input data.

First claim

What is claimed is:

1. A computer-implemented method for efficiently processing data which accounts for both local and global dependencies, the method comprising:
accessing data descriptive of a machine-learned conformer model that comprises one or more conformer blocks, each of the one or more conformer blocks configured to process a block input to generate a block output, each of the one or more conformer blocks comprising:
a first feed-forward block configured to process the block input to generate a first feed-forward output;
a self-attention block configured to perform self-attention to process the first feed-forward output to generate an attention output;
a convolutional block configured to perform convolutions with a convolutional filter to process the attention output of the self-attention block to generate a convolutional output; and
a second feed-forward block configured to process the convolutional output of the convolutional block to generate a second feed-forward output;

obtaining input data, wherein the input data comprises audio data; and
processing the input data with the machine-learned conformer model to generate output data, wherein the output data comprises text data, wherein the machine-learned conformer model comprises the convolutional block that processes outputs of the self-attention block without performing parallel processing with the self-attention block and the convolutional block, and wherein the convolutional block and the self-attention block are between the first feed-forward block and the second feed-forward block.

2. The computer-implemented method of claim 1 , wherein the machine-learned conformer model is an encoder model, and the output data comprises an encoding.

3. The computer-implemented method of claim 1 , wherein each of the first feed-forward block and the second feed-forward block comprises a respective half-step feed-forward block.

4. The computer-implemented method of claim 1 , wherein each of the one or more conformer blocks further comprises a layer normalization block configured to normalize the second feed-forward output to generate the block output.

5. The computer-implemented method of claim 1 , wherein the self-attention block comprises a multi-head self-attention block.

6. The computer-implemented method of claim 1 , wherein the self-attention block comprises a layer normalization block before a multi-head attention with relative positional embedding block.

7. The computer-implemented method of claim 6 , wherein the self-attention block further comprises a dropout block after the multi-head attention with relative positional embedding block.

8. The computer-implemented method of claim 1 , wherein the convolution block comprises a pointwise convolution block followed by a gated linear unit (GLU) activation.

9. The computer-implemented method of claim 1 , wherein the convolution block comprises a 1D depthwise convolution block followed by a Swish activation.

10. The computer-implemented method of claim 1 , wherein the convolution block comprises a layer normalization block, a first pointwise convolution block, a second pointwise convolution block, and a dropout block.

11. The computer-implemented method of claim 1 , wherein the first feed-forward block comprises a first linear layer, a Swish activation, and a second linear layer.

12. The computer-implemented method of claim 1 , wherein the second feed-forward block comprises a first linear layer, a Swish activation, and a second linear layer.

13. The computer-implemented method of claim 1 , wherein the one or more conformer blocks comprise a plurality of conformer blocks stacked in a sequence one after the other.

14. The computer-implemented method of claim 1 , wherein the audio data further comprises spectrograph data descriptive of human speech, and the text data comprises speech recognized data for the human speech.

15. The computer-implemented method of claim 1 , wherein the output data further comprises sound separation data for the audio data.

16. The computer-implemented method of claim 1 , wherein the first feed-forward block, the self-attention block, the convolutional block, and the second feed-forward block each have a respective residual connection.

17. A computing system, comprising:
one or more processors;
one or more non-transitory computer-readable media that collectively store:
a machine-learned conformer model, wherein the machine-learned conformer model comprises:
a first feed-forward block;
a self-attention block;
a convolutional block configured to receive and process an output of the self-attention block; and
a second feed-forward block; and

instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises audio data; and
processing the input data with the machine-learned conformer model to generate output data, wherein the output data comprises text data, wherein the machine-learned conformer model comprises the convolutional block that processes outputs of the self-attention block without performing parallel processing with the self-attention block and the convolutional block, and wherein the convolutional block and the self-attention block are between the first feed-forward block and the second feed-forward block.

18. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises audio data;
processing the input data with a conformer model, wherein the conformer model comprises:
a first feed-forward block;
a self-attention block;

a convolutional block configured to receive and process an output of the self-attention block; and
a second feed-forward block;

wherein the conformer model comprises the convolutional block that processes outputs of the self-attention block without performing parallel processing with the self-attention block and the convolutional block, and wherein the convolutional block and the self-attention block are between the first feed-forward block and the second feed-forward block; and
in response to processing the input data with the conformer model, generating an output data, wherein the output data comprises text data.

Assignees

• Google Llc

Inventors

• Gulati Anmol
• Pang Ruoming
• Parmar Niki
• Yu Jiahui
• Han Wei
• Chiu Chung-Cheng
• Zhang Yu
• Wu Yonghui
• Wang Shibo
• Qin Weikeng
• Zhang Zhengdong

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title

• G06N20/00

  Machine learning · CPC title