Normalizing electronic communications using a vector having a repeating substring as input for a neural network

Patent metadata

Field	Value
Publication number	US-9595002-B2
Application number	US-201615175503-A
Country	US
Kind code	B2
Filing date	Jun 7, 2016
Priority date	May 29, 2015
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

Electronic communications can be normalized using a neural network. For example, a noncanonical communication that includes multiple terms can be received. The noncanonical communication can be preprocessed by (I) generating a vector including multiple characters from a term of the multiple terms; and (II) repeating a substring of the term in the vector such that a last character of the substring is positioned in a last position in the vector. The vector can be transmitted to a neural network configured to receive the vector and generate multiple probabilities based on the vector. A normalized version of the noncanonical communication can be determined using one or more of the multiple probabilities generated by the neural network. Whether the normalized version of the noncanonical communication should be outputted can also be determined using at least one of the multiple probabilities generated by the neural network.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising program code executable by a processor for causing the processor to:
receive an electronic representation of a noncanonical communication, the noncanonical communication including multiple terms;
preprocess the noncanonical communication by:
generating a vector comprising a plurality of characters from a term of the multiple terms, the vector having a predetermined length greater than a length of the term; and
repeating a substring of the term in the vector such that a last character of the substring is positioned in a last position in the vector, wherein the last character of the substring is the same as the last character in the term;

transmit the vector to a neural network comprising at least two bidirectional gated recurrent neural network (BGRNN) layers, the neural network being configured to receive the vector and generate multiple probabilities based on the vector;
receive from the neural network the multiple probabilities generated based on the transmitted vector;
determine a normalized version of the noncanonical communication using one or more of the multiple probabilities received from the neural network; and
determine that the normalized version of the noncanonical communication should be outputted or should not be outputted using at least one of the multiple probabilities received from the neural network.

2. The non-transitory computer readable medium of claim 1 , wherein the neural network comprises:
a character projection layer configured to receive the vector, generate an embedding based on the vector, and transmit the embedding;
a first feed forward neural network configured to receive the embedding, apply matrix operations to the embedding to generate a term embedding, and transmit the term embedding;
a BGRNN that includes the at least two BGRNN layers, the BGRNN configured to receive the term embedding, analyze the noncanonical communication in two directions, and transmit a BGRNN output;
a second feed-forward neural network configured to receive the BGRNN output, apply additional matrix operations to the BGRNN output, and transmit a result of the additional matrix operations; and
a softmax layer configured to receive the result from the second feed-forward neural network and generate the multiple probabilities based on the result of the additional matrix operations.

3. The non-transitory computer readable medium of claim 2 , further comprising program code executable by the processor for causing the processor to preprocess the noncanonical communication by:
replacing each character in the vector with a corresponding index number.

4. The non-transitory computer readable medium of claim 3 , wherein the character projection layer is further configured to:
receive the vector;
determine a plurality of floating point vectors associated with a plurality of index numbers from the vector, each floating point vector corresponding to a particular index number from the vector; and
concatenate the plurality of floating point vectors together into a single vector, the single vector being the embedding.

5. The non-transitory computer readable medium of claim 2 , wherein every respective unit of the first feed-forward neural network only propagates an output value to a subsequent layer of the neural network.

6. The non-transitory computer readable medium of claim 2 , wherein one layer of the at least two BGRNN layers is configured to analyze the noncanonical communication in a forward direction and another layer of the at least two BGRNN layers is configured to analyze the noncanonical communication in a backward direction.

7. The non-transitory computer readable medium of claim 2 , wherein the at least two BGRNN layers are hidden layers, a hidden layer comprising a layer of units between an input layer and an output layer of the neural network; and
wherein the at least two BGRNN layers comprise a plurality of gated recurrent units, each gated recurrent unit of the plurality of gated recurrent units comprising a reset gate and an update gate for controlling an output from the gated recurrent unit.

8. The non-transitory computer readable medium of claim 2 , wherein the softmax layer of the neural network is configured to transform the result of the additional matrix operations into a plurality of values that sum to a total value of one, each value of the plurality of values being a number between zero and one and representing a probability of a particular character being in a particular position in the normalized version of the noncanonical communication; and
wherein determining the normalized version of the noncanonical communication using one or more of the multiple probabilities generated by the neural network comprises:
determining the normalized version of the noncanonical communication based on the plurality of values.

9. The non-transitory computer readable medium of claim 2 , wherein the softmax layer of the neural network is configured to transform the result of the additional matrix operations into a plurality of values that sum to a total value of one, a first value of the plurality of values indicating that the normalized version of the noncanonical communication should be output and a second value of the plurality of values indicating that the normalized version of the noncanonical communication should not be output; and
wherein determining that the normalized version of the noncanonical communication should be outputted or should not be outputted using at least one of the multiple probabilities generated by the neural network comprises:
determining that the normalized version of the noncanonical, communication should be outputted in response to the first value being greater than the second value, and that the normalized version of the noncanonical communication should not be outputted in response to the second value being greater than the first value.

10. The non-transitory computer readable medium of claim 1 , further comprising program code executable by the processor for causing the processor to:
include the normalized version of the noncanonical communication in a data set for use in textual analysis; and
perform textual analysis on the data set to determine one or more trends indicated by the data set.

11. A method comprising:
receiving an electronic representation of a noncanonical communication, the noncanonical communication including multiple terms;
preprocessing the noncanonical communication by:
generating a vector comprising a plurality of characters from a term of the multiple terms, the vector having a predetermined length greater than a length of the term; and
repeating a substring of the term in the vector such that a last character of the substring is positioned in a last position in the vector, wherein the last character of the substring is the same as the last character in the term;

transmitting the vector to a neural network comprising at least two bidirectional gated recurrent neural network (BGRNN) layers, the neural network being configured to receive the vector and generate multiple probabilities based on the vector;
receiving from the neural network the multiple probabilities generated based on the transmitted vector;
determining a normalized version of the noncanonical communication using one or more of the multiple probabilities received from the neural network; and
determining that the normalized version of the noncanonical communication should be outputted or should not be outputted using at least one of the multiple probabilities received from the neural network.

12. The method of claim 11 , wherein the neural ne

Assignees

• Sas Inst Inc

Inventors

• Leeman-Munk Samuel Paul
• Cox James Allen

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/0442 Primary

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0499

  Feedforward networks · CPC title