Analyzing sequences of interactions using a neural network with attention mechanism

Patent metadata

Field	Value
Publication number	US-11714997-B2
Application number	US-202117204578-A
Country	US
Kind code	B2
Filing date	Mar 17, 2021
Priority date	Mar 17, 2021
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

Users interact with a computer system, which collects data about individual interactions the users have had with the computer system. The users are sorted into one of a first group or a second group. The computer system generates respective user sequence models for the users using information representing the individual interactions. The computer system analyzes the respective user sequence models using a recurrent neural network with an attention mechanism, which produces respective vectors corresponding to the user sequence models. Individual values in the vectors represent respective individual interactions by a given user and correspond to an amount of correlation between the respective individual interactions and the sorting of the given user into the first group or the second group. The computer system identifies a particular type of interaction that is correlated to users being sorted into the first group by analyzing the respective vectors.

First claim

What is claimed is:

1. A method comprising:
accessing, with a computer system, a plurality of data stores storing information representing individual interactions with a plurality of users, wherein individual users of the plurality of users have been sorted into one of a first group or a second group;
generating, by the computer system, respective user sequence models for the plurality of users using the information representing individual interactions with the plurality of users;
analyzing, by the computer system, the respective user sequence models using a recurrent neural network with an attention mechanism implemented as one or more attention layers, wherein the analyzing produces respective vectors corresponding to the respective user sequence models, wherein the respective vectors include respective sequences of attention weights, wherein a given sequence of attention weights for a given vector (a) represents respective individual interactions by a given user and (b) corresponds to an amount of correlation between the respective individual interactions and the sorting of the given user into the first group or the second group; and
identifying, by the computer system analyzing the respective vectors, a particular type of interaction that is correlated to users being sorted into the first group.

2. The method of claim 1 ,
wherein analyzing the respective vectors includes comparing attention weights for a given type of interaction for users sorted into the first group with attention weights for the given type of interaction for users sorted into the second group; and
wherein the particular type of interaction is identified based on the comparing.

3. The method of claim 1 ,
wherein analyzing the respective vectors includes performing a statistical analysis of the attention weights of the respective vectors corresponding to users grouped in the first group, and
wherein the particular type of interaction is identified based on the statistical analysis.

4. The method of claim 3 , wherein the statistical analysis includes:
dividing a mean of the attention weights corresponding to the particular type of interaction with users grouped in the first group by a mean of the attention weights corresponding to the particular type of interaction with users grouped in the second group; and
identifying the particular type of interaction based on a quotient for the particular type of interaction having the maximum or minimum value from among quotients corresponding to other types of interaction.

5. The method of claim 3 , wherein the statistical analysis includes correlating the attention weights for the particular type of interaction with positions of representations of individual interactions of the particular type of interaction in the respective user sequence models.

6. The method of claim 1 ,
wherein a sum of the attention weights of the given vector equals the sums of the attention weights of the other respective vectors; and
wherein the attention weights range between 0 and 1.

7. The method of claim 1 , further comprising:
determining a hyperparameter for the recurrent neural network, wherein the number of individual interactions to represent in the respective user sequence models is based on the hyperparameter;
wherein the number of attention weights in the respective vectors is based on the hyperparameter.

8. The method of claim 7 , further comprising:
generating, by a computer system, training user sequence models using a second plurality of users, wherein the second plurality of users are sorted into one of the first group or the second group, wherein the generating includes:
for a first user who has had more individual interactions than the hyperparameter, generating a first user sequence model based on the first X individual interactions, wherein X is equal to the hyperparameter, and
for a second user who has had fewer individual interactions than the hyperparameter, excluding the second user from the second plurality of users such that a training user sequence model is not generated for the second user; and

training the recurrent neural network using the training user sequence models.

9. The method of claim 1 , further comprising:
accessing, with the computer system, a plurality of data stores storing information corresponding to the plurality of users;
wherein generating the respective user sequence models includes using information accessed from the plurality of data stores such that the respective user sequence models for one or more of the plurality of users are generated using information accessed from separate data stores.

10. A non-transitory, computer-readable medium storing instructions that when executed by a computer system cause the computer system to perform operations comprising:
accessing, with a computer system, a plurality of data stores storing information representing individual interactions with a plurality of users, wherein individual users of the plurality of users have been sorted into one of a first group or a second group;
generating respective user sequence models for a plurality of users, wherein entries in the respective user sequence models represent individual interactions by respective users;
analyzing the respective user sequence models using a recurrent neural network with an attention mechanism implemented as one or more attention layers, wherein the analyzing produces respective vectors corresponding to the respective user sequence models, wherein the respective vectors include respective sequences of attention weights calculated by the recurrent neural network that represent respective individual interactions by the respective users, wherein the attention weights correspond to an amount of correlation between the respective individual interactions and the sorting of the given user into the first group or the second group; and
identifying, based on analyzing the respective vectors, a first particular type of interaction that is correlated to users belonging to the first group and a second particular type of interaction that is correlated to users belonging to the second group.

11. The computer-readable medium of claim 10 ,
wherein the first particular type of interaction contributed the most to users being sorted into the first group; and
wherein the second particular type of interaction contributed the most to users being sorted into the second group.

12. The computer-readable medium of claim 10 ,
wherein analyzing the respective vectors includes comparing:
values calculated using respective sequences of attention weights for respective vectors of users sorted in the first group, and
values calculated using respective sequences of attention weights for respective vectors of users sorted in the second group.

13. The computer-readable medium of claim 10 , wherein the first group corresponds to users who have reported fraudulent account activity and the second group corresponds to users who have not reported fraudulent account activity.

14. The computer-readable medium of claim 13 , wherein the operations further comprise:
based on identifying the first particular type of interaction, adjusting one or more security parameters for the computer system corresponding to the first particular type of interaction.

15. The computer-readable medium of claim 10 , wherein the recurrent neural network is implemented by a long short-term memory and the attention mechanism is implemented using one or more attention layers of the long short-term memory.

16. A method comprising:
accessing, with a computer system, a plurality of data stor

Assignees

• Paypal Inc

Inventors

• Saleh Moein
• Poletti Chiara
• He Shiying
• Oppenheim Hagar
• Ji Xing

Classifications

• G06N3/049 Primary

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06Q20/4016

  involving fraud or risk level assessment in transaction processing · CPC title

• H04L67/306 Primary

  User profiles · CPC title

• G06N3/045

  Combinations of networks · CPC title