Differentiating between good and bad content in a user-provided content system

Patent metadata

Field	Value
Publication number	US-9356920-B2
Application number	US-201414332042-A
Country	US
Kind code	B2
Filing date	Jul 15, 2014
Priority date	Nov 15, 2010
Publication date	May 31, 2016
Grant date	May 31, 2016

Abstract

A system differentiates good content from bad content in a user-provided content system. Messages are analyzed for features that characterize messages. A feature may occur in one or more messages. A feature that has more than a threshold number of occurrences in messages in a time interval is identified for further analysis. Enhanced authentication is requested from senders of the messages with occurrences of the identified feature. Based on the rate at which senders of the messages pass authentication, the content associated with the message is determined to be good content or bad content. Subsequent messages are blocked or successfully delivered based on whether features occurring in the messages are indicative of good content or bad content.

First claim

What is claimed is:

1. A computer implemented method comprising:
receiving, at a social networking system, a plurality of posts from a plurality of users of a user-provided content system, each post for display to one or more other users of the system;
identifying a set of features associated with the received posts, each feature being extracted from content in one or more posts and not in a black list maintained by the social networking system;
identifying a feature of the set of features based on a rate of occurrence of the feature in the plurality of posts and one or more types of actions associated with the feature;
determining a level of maliciousness of the identified feature based on the one or more types of actions associated with the identified feature;
requesting enhanced authentication from one or more users who generated the posts in which the identified feature occurred based on the determined level of maliciousness associated with the identified feature;
determining that the identified feature is indicative of bad content based on a rate of users who passed the authentication;
adding the identified feature to the black list; and
blocking from display subsequent posts received in which the identified feature occurs.

2. The computer implemented method of claim 1 , wherein the identified feature is indicative of bad content if the rate of users who passed the authentication is less than a threshold rate.

3. The computer implemented method of claim 1 , further comprising:
responsive to determining that the identified feature is indicative of bad content, blocking display of the post.

4. The computer implemented method of claim 1 , wherein the rate of users required to pass authentication is determined based on a measure of risk associated with the set of features.

5. The computer implemented method of claim 1 , wherein the rate of users required to pass authentication is determined based on the rate of occurrence of the identified feature in the plurality of posts received.

6. The computer implemented method of claim 1 , further comprising:
determining a second rate of users who passed the authentication;
determining that the identified feature is indicative of good content based on the second rate;
removing the identified feature from the black list; and
unblocking from display subsequent posts received in which the identified feature occurs.

7. The computer implemented method of claim 1 , wherein the identified feature is indicative of good content if the second rate of users who passed the authentication exceeds a threshold rate.

8. The computer implemented method of claim 1 , further comprising:
responsive to determining that the identified feature is indicative of good content, unblocking display of the post.

9. The computer implemented method of claim 1 , wherein enhanced authentication comprises sending a challenge question.

10. The computer implemented method of claim 1 , further comprising determining a metric describing a level of risk associated with the identified feature based on the rate of users passing authentication.

11. The computer implemented method of claim 1 , wherein the identified feature comprises a uniform record locator address included in a post.

12. The computer implemented method of claim 1 , wherein the identified feature comprises a network address of a generator of a post.

13. The computer implemented method of claim 1 , wherein the identified feature comprises a netblock associated with a generator of a post.

14. The computer implemented method of claim 1 , wherein the identified feature comprises a domain of a generator of a post.

15. The computer implemented method of claim 1 , wherein the identified feature comprises a regular expression based on terms included in a post.

16. The computer implemented method of claim 1 , wherein the identified feature comprises a time of day associated with transmission of a post.

17. The computer implemented method of claim 1 , wherein the identified feature comprises information identifying an application executing on a client device sending a post.

18. The computer implemented method of claim 17 , wherein the application executing on a client device is a browser application and the information identifying the browser is a hash value based on browser settings.

19. The computer implemented method of claim 1 , wherein the identified feature comprises a netblock of a generator of a message and information describing an interface used to communicate the message.

20. A non-transitory computer-readable storage medium comprising instructions executable by a processor, the instructions for:
receiving, at a social networking system, a plurality of posts from a plurality of users of a user-provided content system, each post for display to one or more other users of the system;
identifying a set of features associated with the received posts, each feature being extracted from content in one or more posts and not in a black list maintained by the social networking system;
identifying a feature of the set of features based on a rate of occurrence of the feature in the plurality of posts and one or more types of actions associated with the feature;
determining a level of maliciousness of the identified feature based on the one or more types of actions associated with the identified feature;
requesting enhanced authentication from one or more users who generated the posts in which the identified feature occurred based on the determined level of maliciousness associated with the identified feature;
determining that the identified feature is indicative of bad content based on a rate of users who passed the authentication;
adding the identified feature to the black list; and
blocking from display subsequent posts received in which the identified feature occurs.

Assignees

• Facebook Inc

Inventors

• Stein Christopher Alexander
• Popov Lev Timourovich
• Stratford Evan

Classifications

• G06F21/552 Primary

  involving long-term monitoring or reporting · CPC title

• H04L51/32

  Electricity · mapped topic

• H04L12/585

  Electricity · mapped topic

• H04L63/08 Primary

  for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32) · CPC title

• G06F21/566

  Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities · CPC title