Electronic message composition support method and apparatus

Patent metadata

Field	Value
Publication number	US-11265271-B2
Application number	US-201916258783-A
Country	US
Kind code	B2
Filing date	Jan 28, 2019
Priority date	Mar 3, 2016
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

An electronic message composition support system, method and architecture including machine learning and natural language processing techniques for extending message composition capability and support and to provide feedback to a user regarding an error, condition, etc., detected in the user's message before the user sends the message, e.g., while the user is composing the message using a messaging application's user interface.

First claim

The invention claimed is:

1. A method comprising:
obtaining, by a computing device, training data comprising a number of training examples, each training example comprising a set of features and having a corresponding label indicating whether or not the example is a positive example of a condition meriting correction;
training, by the computing device and using a machine learning algorithm, a linter using the training data, the linter for use in detecting the condition meriting correction in connection with electronic message input, of a user, the training comprising providing each training example and corresponding label as input to the machine learning algorithm to train the linter to recognize the condition meriting correction, the training comprising recognizing whether or not a feature, of the set of features, is a predictive feature of the condition meriting correction;
generating, by the computing device, feature set input comprising the set of features generated using the electronic message input, of the user, the feature set input for use by the linter in detecting the condition meriting correction;
making a determination, by the computing device and using the linter, that the condition meriting correction exists with the electronic message input, the condition comprising a stylistic condition, the determining comprising:
providing the feature set input as input to the linter trained using the machine learning algorithm and each of the training examples comprising the set of features; and
receiving output, from the linter, the output comprising a probability being used to make the determination identifying a style;

generating, by the computing device, feedback indicative of the stylistic condition in accordance with the determination, the feedback specifying both the style determined by the linter and a recommended style to use instead, the feedback suggesting that, prior to sending, the user change to the different style for the electronic message; and
displaying, via the computing device, the feedback to the user.

2. The method of claim 1 , further comprising:
testing, by the computing device, the linter using a number of test data examples comprising a number of examples from the training data excluding the corresponding label, the testing comprising providing each test data example to the linter, the testing indicating a level of accuracy of the linter in detecting the condition.

3. The method of claim 1 , further comprising training a confused-word linter for use with a pair of confused words, the number of training examples, of the training data for training the confused-word linter, comprising a first set of training examples and a second set of training examples, each example in the first set comprising the set of features and having a label identifying a first word in the confused word pair and each example in the second set comprising the set of features and having a label identifying a second word in the confused word pair.

4. The method of claim 3 , the set of features comprising a number of n-gram features, in a case of the first set of training examples, an n-gram feature, of the number of n-gram features, comprising a number of words to the left and a number of words to the right of the first word in the confused-word pair, in the case of the second set of training examples, an n-gram feature, of the number of n-gram features, comprising a number of words to the left and a number of words to the right of the second word in the confused-word pair.

5. The method of claim 3 :
the providing further comprising providing, as the feature set input corresponding to the electronic message input, at least one feature comprising an n-gram determined from the electronic message input; and
the receiving further comprising receiving output comprising a prediction indicating which word of the confused-word pair is correctly used in the electronic message input;
the generating feedback further comprising generating feedback in a case that the electronic message input includes an incorrectly-used word based on the received output from the linter.

6. The method of claim 1 , the linter is a stylistic linter, for each training example, of the number of training examples, the set of features comprising at least one n-gram, each n-gram being associated with a label indicating whether or not the example is a positive stylistic example.

7. The method of claim 6 , n is variable, such that a value of n for one n-gram differs from the value of n for at least one other n-gram.

8. The method of claim 6 :
the providing further comprising providing, as the feature set input corresponding to the electronic message input, at least n-gram determined from the electronic message input; and
the receiving further comprising receiving output comprising a prediction indicating whether or not a stylistic condition exists in the electronic message input.

9. The method of claim 6 , the stylistic linter is selected from the group consisting of a hate linter, a sentiment linter, a satire linter, sarcasm linter, or an abusive language linter.

10. The method of claim 1 , the linter is a formality linter for use in identifying a mismatch between a determined level of formality of content of the electronic message input and a desired level of formality, for each training example, of the number of training examples, the set of features comprising a number of natural language processing features representing content of the training example and a corresponding label indicating a level of formality for the training example.

11. The method of claim 10 , for each training example, of the number of training examples, the set of features comprising a number of case features, a number of punctuation features, a number of readability features, a number of subjectivity features, a number of parts-of-speech (POS) features, a number of dependency features, a number of entity features and a number of word2vec features.

12. The method of claim 10 :
the providing further comprising providing, as the feature set input corresponding to the electronic message input, a number of features representing content of at least a portion of the electronic message input determined from the electronic message input; and
the receiving further comprising receiving output comprising a prediction indicating level of formality corresponding to the electronic message input;
the generating feedback further comprising generating feedback indicating that the level of formality and the desired level of formality are a mismatch.

13. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
obtaining training data comprising a number of training examples, each training example comprising a set of features and having a corresponding label indicating whether or not the example is a positive example of a condition meriting correction;
training, using a machine learning algorithm, a linter using the training data, the linter for use in detecting the condition meriting correction in connection with electronic message input, of a user, the training comprising providing each training example and corresponding label as input to the machine learning algorithm to train the linter to recognize the condition meriting correction, the training comprising recognizing whether or not a feature, of the set of features, is a predictive feature of the condition meriting correction;
generating feature set input comprising the set of features generated using the electronic message input, o

Assignees

• Verizon Media Inc
• Yahoo Assets Llc

Inventors

• Tetreault Joel
• Pappu Aasish
• Liberty Edo
• Cao Liangliang
• Liu Meizhu
• Pavlick Ellie
• Tsur Gilad
• Maarek Yoelle

Classifications

• G06F40/268

  Morphological analysis · CPC title

• H04L67/02

  based on web technology, e.g. hypertext transfer protocol [HTTP] · CPC title

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• H04L51/046 Primary

  Interoperability with other network applications or services · CPC title

• G06F40/253

  Grammatical analysis; Style critique · CPC title