Method and apparatus with key-value coupling

Patent metadata

Field	Value
Publication number	US-2020320408-A1
Application number	US-201916535502-A
Country	US
Kind code	A1
Filing date	Aug 8, 2019
Priority date	Apr 2, 2019
Publication date	Oct 8, 2020
Grant date	—

Abstract

A processor-implemented method of implementing an attention mechanism in a neural network includes obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of an attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

First claim

What is claimed is:

1 . A processor-implemented method of implementing an attention mechanism in a neural network, the method comprising:
obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of an attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

2 . The method of claim 1 , wherein the obtaining comprises:
determining the new key data by applying the first nonlinear transformation to the key data; and determining the key-value coupling data based on an operation between the value data and the new key data.

3 . The method of claim 1 , wherein
the new key data includes a first new key, and the value data includes a first value corresponding to the first new key, and the key-value coupling data includes a single item of aggregated data determined based an operation between the first new key and the first value with respect to a first key-value pair of the first new key and the first value.

4 . The method of claim 1 , wherein either one or both of the first nonlinear transformation and the second nonlinear transformation uses either one or both of a sine function and a cosine function as a nonlinear factor.

5 . The method of claim 1 , wherein the first nonlinear transformation and the second nonlinear transformation use the same function.

6 . The method of claim 1 , wherein the key-value coupling data is fixed based on an operation between the new key data and the value data, and
the output data of the attention layer is determined based on an operation between the new query data and the fixed key-value coupling data.

7 . The method of claim 6 , wherein the key-value coupling data is fixed by being determined, independent of the query data, based on the operation between the new key data and the value data.

8 . The method of claim 1 , wherein an operation between the new key data and the new query data corresponds to a similarity between the key data and the query data.

9 . The method of claim 1 , wherein the determining of the output data of the attention layer comprises normalizing a result of the operation between the new query data and the key-value coupling data.

10 . The method of claim 1 , further comprising performing an inference operation using the neural network based on the output data of the attention layer, wherein the neural network includes additional trained layers.

11 . The method of claim 1 , further comprising outputting an image recognition result for the input data by applying the output data of the attention layer to the neural network.

12 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .

13 . A processor-implemented nonlocal filtering method, comprising:
obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data corresponding to patches in an input image, and value data of representative pixels in the patches; determining new query data by applying a second nonlinear transformation to query data corresponding to a target patch among the patches; and determining output data for denoising of a representative pixel in the target patch, based on an operation between the new query data and the key-value coupling data.

14 . The method of claim 13 , wherein the representative pixels in the patches are center pixels in the patches, and the representative pixel in the target patch is a center pixel in the target patch.

15 . The method of claim 13 , wherein the obtaining comprises:
determining the new key data by applying the first nonlinear transformation to the key data; and determining the key-value coupling data based on an operation between the value data and the new key data.

16 . The method of claim 13 , wherein
the new key data includes a first new key, and the value data includes a first value corresponding to the first new key, and the key-value coupling data includes a single item of aggregated data determined based on an operation between the first new key and the first value with respect to a first key-value pair of the first new key and the first value.

17 . The method of claim 13 , wherein either one or both of the first nonlinear transformation and the second nonlinear transformation uses either one or both of a sine function and a cosine function as a nonlinear factor.

18 . The method of claim 13 , wherein the first nonlinear transformation and the second nonlinear transformation use the same function.

19 . The method of claim 13 , wherein an operation between the new key data and the new query data corresponds to a similarity between the key data and the query data.

20 . The method of claim 13 , further comprising denoising the representative pixel in the target patch based on the output data.

21 . A processor-implemented method of implementing a neural network, the method comprising:
performing an inference related to input data of the neural network using a plurality of layers in the neural network, wherein at least one of the plurality of layers in the neural network uses either one or both of a sine function and a cosine function to obtain a nonlinearity.

22 . The method of claim 21 , wherein the at least one layer is a respective attention layer that performs a corresponding attention mechanism.

23 . The method of claim 22 , wherein the performing comprises:
obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of the attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

24 . The method of claim 23 , wherein either one or both of the first nonlinear transformation and the second nonlinear transformation uses either one or both of the sine function and the cosine function.

25 . A processor-implemented method of implementing an attention mechanism in a neural network, the method comprising:
obtaining fixed key-value coupling data determined, independently of input query data of an attention layer, based on key data of the attention layer and value data corresponding to the key data; determining new query data based on input query data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

26 . The method of claim 24 , wherein
the new key data is determined by applying a first nonlinear transformation to the key data, the key-value coupling data is determined based on an operation between the value data and the new key data, and the determining of the new query data

Assignees

• Samsung Electronics Co Ltd

Inventors

• Chang Hyun Sung
• Lee Wonhee
• Kang Nahyup

Classifications

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06F18/21355

  nonlinear criteria, e.g. embedding a manifold in a Euclidean space · CPC title