Online Active Learning In User-Generated Content Streams

Patent metadata

Field	Value
Publication number	US-2018255012-A1
Application number	US-201815973130-A
Country	US
Kind code	A1
Filing date	May 7, 2018
Priority date	Oct 26, 2011
Publication date	Sep 6, 2018
Grant date	—

Abstract

Software for online active learning receives content posted to an online stream at a website. The software converts the content into an elemental representation and inputs the elemental representation into a probit model to obtain a predictive probability that the content is abusive. The software also calculates an importance weight based on the elemental representation. And the software updates the probit model using the content, the importance weight, and an acquired label if a condition is met. The condition depends on an instrumental distribution. The software removes the content from the online stream if a condition is met. The condition depends on the predictive probability, if an acquired label is unavailable.

First claim

What is claimed is:

1 . A method for delivering content for display on client devices, comprising the operations of:
receiving, at one or more servers, content that includes a plurality of user generated content (UGC) items, at least a portion of which is abusive; processing, at the one or more servers, the plurality of UGC items using a predictive model to obtain a predictive probability of whether each of the plurality of UGC items is abusive or not abusive; selecting, based on the predictive probability, specific ones of the plurality of UGC items for which to request a label; updating, in response to receipt of one or more labels, the predictive model using the one or more labels; applying the predictive model having been updated to the plurality of UGC items for removing the portion of the plurality of UGC items that is abusive from the content; and serving, to a client device, the content without the portion of the plurality of the UGC items that is abusive.

2 . The method of claim 1 , wherein the selecting specific ones of the plurality of UGC items further includes:
creating a probabilistic queue for delivering the plurality of UGC items to a human labeler, wherein an order of the probabilistic queue is dependent on the predictive probability.

3 . The method of claim 2 , wherein UGC items that are associated with a predictive probability of being abusive have a placement within the probabilistic queue that is ahead of UGC items that are associated with a predictive probability of being not abusive.

4 . The method of claim 1 , wherein the processing the plurality of the UGC items is further configured to obtain an importance weight for each of the plurality of UGC items, and wherein the updating the predictive model includes adjusting the importance weight using the one or more labels.

5 . The method of claim 1 , wherein said selecting specific ones of the plurality of UGC items for which to request the label is further based on an instrumental distribution.

6 . The method of claim 1 , wherein the UGC items include text, the text is processed as an elemental representation.

7 . The method of claim 1 , wherein the UGC items include an image, the image is processed using a bag of features model.

8 . The method of claim 1 , wherein the portion of the plurality of UGC items that is abusive includes one or more of spam, fraudulent offers, illegal offers, offensive language, threatening language, and treasonous language.

9 . The method of claim 1 , wherein the predictive model includes a memory loss factor, the memory loss factor enables the predictive model to adapt to concept drift of UGC items.

10 . A computer-implemented method, comprising the operations of:
receiving, at one or more servers, a plurality of user generated content (UGC) items that has been posted to an online stream at a website, at least of portion of which is abusive; processing, at the one or more servers, the plurality of UGC items using a predictive model to obtain a predictive probability of whether each of the plurality of UGC items is abusive or not abusive, the predictive probability is used for selecting specific ones of the plurality of UGC items associated with a label and the predictive model is updated using one or more labels when received, wherein said processing removes the portion of the plurality UGC items based on the predictive probability; and delivering, to a client device when the client device accesses the website, the plurality of UGC items without the portion of UGC items that is abusive.

11 . The computer-implemented method of claim 10 , wherein said selecting specific ones of the plurality of UGC items associated with the label further includes:
creating a queue for delivering selected ones of the plurality of UGC items to a human labeler that generates the label and other labels responsive to label requests, an order of the queue depends upon the predictive probability; wherein the specific ones of the plurality of UGC items become associated with the label once the label is generated.

12 . The computer-implemented method of claim 11 , wherein the order is such that UGC items having a predictive probability of being abusive are placed ahead of UGC items having a predictive probability of being not abusive within the queue.

13 . The computer-implemented method of claim 10 , wherein the processing the plurality of UGC items is further configured to obtain an importance weight for each of the plurality of UGC items, and wherein the predictive model is additionally updated by adjusting the importance weight using the one or more labels.

14 . The computer-implemented method of claim 10 , wherein said selecting specific ones of the plurality of UGC items associated with a label is further based on an instrumental distribution.

15 . The computer-implemented method of claim 10 , wherein at least a portion of the UGC items includes text, the text is processed as elemental representations.

16 . The computer-implemented method of claim 10 , wherein at least a portion of the UGC items includes an image, the image is processed using a bag of features model.

17 . The computer-implemented method of claim 10 , wherein the portion of the plurality of UGC items that is abusive includes one or more of span, fraudulent offers, illegal offers, offensive language, threatening language, and treasonous language.

18 . The computer-implemented method of claim 10 , wherein the predictive model includes a memory loss factor that enables the predictive model to adapt to concept drift of UGC items.

19 . A non-transitory computer-readable medium storing a computer program executable by a processor-based system, comprising:
program instructions for receiving, at one or more servers, a plurality of user generated content (UGC) items that has been posted to an online stream at a website, at least of portion of which is abusive; program instructions for processing, at the one or more servers, the plurality of UGC items using a predictive model to obtain a predictive probability of whether each of the plurality of UGC items is abusive or not abusive, the predictive probability is used for selecting specific ones of the plurality of UGC items associated with a label and the predictive model is updated using one or more labels when received, wherein said processing removes the portion of the plurality UGC items based on the predictive probability; and program instructions for delivering, to a client device when the client device accesses the website, the plurality of UGC items without the portion of UGC items that is abusive.

20 . The non-transitory computer-readable medium of claim 19 , wherein said selecting the specific ones of the plurality of UGC items associated with the label further includes:
creating a queue for delivering selected ones of the plurality of UGC items to a human labeler that generates the label and other labels responsive to label requests, wherein an order of the queue depends from the predictive probability such that UGC items having a predictive probability of being abusive are placed ahead of UGC items having a predictive probability of being not abusive within the queue; wherein the specific ones of the plurality of UGC items become associated with the label once the label is generated.

Assignees

• Oath Inc

Inventors

• Chu Wei
• Zinkevich Martin
• Li Lihong
• Thomas Achint Oommen
• Tseng Belle

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06Q50/20 Primary

  Education · CPC title

• G06F11/00

  Error detection; Error correction; Monitoring (error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00) · CPC title

• G06N7/00

  Computing arrangements based on specific mathematical models · CPC title

• G06F15/16

  Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs {(coordinating program control therefor G06F9/52; in regulating and control system G05B)} · CPC title