Detecting content items in violation of an online system policy using semantic vectors

Patent metadata

Field	Value
Publication number	US-11195099-B2
Application number	US-201715694321-A
Country	US
Kind code	B2
Filing date	Sep 1, 2017
Priority date	Sep 1, 2017
Publication date	Dec 7, 2021
Grant date	Dec 7, 2021

Abstract

A content review system for an online system automatically determines if received content items to be displayed to users violate any policies of the online system. The content review system generates a semantic vector representing the semantic features of a content item, for example, using a neural network. By comparing the semantic vector for the content item with semantic vectors of content items previously determined to violate one or more policies, the content review system determines whether the content item also violates one or more policies. The content review system may also maintain templates corresponding to portions of semantic vectors shared by multiple content items. An analysis of historical content items that conform to the template is performed to determine a probability that received content items that conform to the template violate a policy.

First claim

What is claimed is:

1. A method comprising:
receiving, by an online system associated with a plurality of policies, a request to determine whether an input content item violates any of the plurality of policies, each policy indicating a type of content item considered suitable or unsuitable for presenting to users of the online system;
accessing one or more data stores storing, for each of the plurality of policies, a respective set of semantic vectors corresponding to content items previously determined to violate the policy;
determining a semantic vector of the input content item, by:
receiving the input content item as input at a deep neural network having a plurality of layers;
determining the semantic vector representing the input content item based on the output of a hidden layer of the deep neural network;

determining a plurality of distance metric values, each corresponding to a distance between the semantic vector of the input content item and one or more semantic vectors of the set of semantic vectors corresponding to a respective policy of the plurality of policies; and
responsive to determining that a distance metric value of the plurality of distance metric values associated with a particular policy of the plurality of policies is below a first threshold value indicating a distance at which a likelihood that the input content item violates the particular policy is at least a first threshold probability, automatically withholding the content item from users of the online system.

2. The method of claim 1 , wherein the semantic vector representing the input content item is obtained by applying a hash function to the output of the hidden layer of the neural network.

3. The method of claim 1 , wherein the neural network is configured to output one or more scores indicating a probability that the input content item contains one or more corresponding semantic features.

4. The method of claim 1 , further comprising:
receiving a request to determine whether a second input content item violates any of the plurality of policies;
determining a second plurality of distance metric values, each corresponding to a distance between a semantic vector of the second input content item and one or more semantic vectors of the set of semantic vectors corresponding to a respective policy of the plurality of policies;
determining that a distance metric value of the second plurality distance metric values associated with the particular policy is above a second threshold value, determining that the second input content item does not violate the particular policy; and
responsive to determining that the second content item does not violate any of the policies of the plurality of policies, sending the second content item to the users of the online system.

5. The method of claim 4 , wherein the second threshold value is greater than the first threshold value, and indicates a distance at which a likelihood that the second input content item violates the particular is at least a second threshold probability that is lower than the first threshold probability.

6. The method of claim 1 , further comprising:
responsive to determining that the distance metric value is above the first threshold value but below a second threshold value, flagging the content item for review.

7. The method of claim 1 , further comprising:
responsive to determining that the distance metric value is below the first threshold value, storing the semantic vector of the input content item as part of the set of semantic vectors.

8. The method of claim 1 , wherein the semantic vector corresponds to a binary hash value, and wherein the distance metric value corresponds to a hamming distance.

9. A method comprising:
receiving, by an online system associated with a plurality of policies, a request to determine whether an input content item violates any of the plurality of policies, each policy indicating a type of content item considered suitable or unsuitable for presenting to users of the online system;
accessing one or more data stores storing, for each of the plurality of policies, a respective set of semantic vectors corresponding to content items previously determined to violate the policy;
determining a semantic vector of the input content item using a deep neural network having a plurality of layers;
determining a plurality of distance metric values, each corresponding to a distance between the semantic vector of the input content item and one or more semantic vectors of the set of semantic vectors corresponding to a respective policy of the plurality of policies; and
responsive to determining that a distance metric value of the plurality of distance metric values associated with a particular policy of the plurality of policies is below a first threshold value indicating a distance at which a likelihood that the input content item violates the particular policy is at least a first threshold probability, automatically withholding the content item from users of the online system.

10. The method of claim 9 , further comprising:
receiving a request to determine whether a second input content item violates any of the plurality of policies;
determining a second plurality of distance metric values, each corresponding to a distance between a semantic vector of the second input content item and one or more semantic vectors of the set of semantic vectors corresponding to a respective policy of the plurality of policies;
determining that a distance metric value of the second plurality distance metric values associated with the particular policy is above a second threshold value, determining that the second input content item does not violate the particular policy; and
responsive to determining that the second content item does not violate any of the policies of the plurality of policies, sending the second content item to the users of the online system.

11. The method of claim 9 , further comprising:
responsive to determining that the distance metric value is above the first threshold value but below a second threshold value, flagging the content item for review.

12. The method of claim 9 , further comprising:
responsive to determining that the distance metric value is below the first threshold value, storing the semantic vector of the input content item as part of the set of semantic vectors.

13. The method of claim 9 , wherein the semantic vector corresponds to an embedding associated with the input content item obtained using the deep neural network.

14. The method of claim 9 , wherein the semantic vector corresponds to a binary hash value, and wherein the distance metric value corresponds to a hamming distance.

15. The method of claim 9 , wherein the input content item comprises an image.

16. A computer readable non-transitory storage medium, storing instructions for:
receiving, by an online system associated with a plurality of policies, a request to determine whether an input content item violates any of the plurality of policies, each policy indicating a type of content item considered suitable or unsuitable for presenting to users of the online system;
accessing one or more data stores storing, for each of the plurality of policies, a respective set of semantic vectors corresponding to content items previously determined to violate the policy;
determining a semantic vector of the input content item using a deep neural network having a plurality of layers;
determining a plurality of distance metric values, each corresponding to a distance between the semantic vector of the input content item and one or more semantic

Assignees

• Facebook Inc

Inventors

• Luo Enming
• Mu Yang
• Strauss Emanuel Alexandre
• Zhang Taiyuan
• Olmedilla De La Calle Daniel

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title