Framework for evaluating targeting models

Patent metadata

Field	Value
Publication number	US-10127573-B2
Application number	US-201514940794-A
Country	US
Kind code	B2
Filing date	Nov 13, 2015
Priority date	Nov 13, 2015
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

An online system predicts, using a first targeting model, a first group of users as candidates to be in a targeting cluster, and predicts, using a second targeting model, a second group of users as candidates to be in the targeting cluster. The online system determines a first set of users that are not part of the first group of users, and a second set of users that are not part of the second group of users, and provides surveys to the first and second set of users. The online system determines a first subgroup of the first group of users and a second subgroup of the second group of users, and provides an ad preferences tool to the first subgroup and the second subgroup. The online system scores the first and second targeting models based in part on responses to the surveys and/or the ad preferences tools.

First claim

What is claimed is:

1. A method comprising:
retrieving a first targeting model and a second targeting model for targeting content to users of an online system, the first targeting model and the second targeting model including a collection of rules that control how characteristics are mapped to users of the online system;
predicting, using the first targeting model, a first group of users of the online system as candidates to be in a targeting cluster associated with a characteristic of interest;
predicting, using the second targeting model, a second group of users of the online system as candidates to be in the targeting cluster;
determining a first set of users of the online system that are not part of the first group of users, and a second set of users of the online system that are not part of the second group of users;
providing surveys to the first set of users and the second set of users, the survey asking whether a user should be associated with the characteristic of interest, the surveys testing whether the first and second targeting models correctly predicted that the user should not be included in the targeting cluster associated with the characteristic of interest;
determining a first subgroup of the first group of users and a second subgroup of the second group of users, the first subgroup and second subgroup composed of users that are associated with the characteristic of interest;
providing a preferences tool to the first subgroup and the second subgroup, the preferences tool prompting a viewing user to modify preferences listed for the viewing user, the preferences tool testing whether the first and second targeting models correctly predicted that the viewing user should be included in the targeting cluster associated with the characteristic of interest; and
scoring the first targeting model and the second targeting model based at least in part on one or more responses to the surveys and one or more responses to the preferences tool, the score measuring precision and recall of the first and second targeting models.

2. The method of claim 1 , further comprising:
selecting the first targeting model as a better performing model based at least in part on the scoring;
retrieving an active targeting model, the active targeting model defining how the online system selects targeting attributes for users of the online system;
predicting, using the active targeting model, a third group of users of the online system as candidates to be in the targeting cluster;
selecting a content item to provide to the first group of users and to the third group of users based in part on targeting criteria associated with the content item matching the characteristic of interest; and
providing the selected content item to user devices associated with the first group of users and the third group of users.

3. The method of claim 2 , further comprising:
determining a metric to evaluate performance of the first targeting model and the active targeting model based in part on the targeting criteria associated with the selected content item;
scoring the first targeting model and the active targeting model using the determined metric; and
based in part on the scoring, selecting as a new active targeting model a targeting model selected from a group consisting of the first targeting model and the active targeting model.

4. The method of claim 1 , wherein scoring the first targeting model and the second targeting model based at least in part on one or more responses to the surveys and one or more responses to the preferences tool, the score measuring precision and recall of the first and second targeting models, comprises:
determining a total number of false negatives and a total number of true negatives using feedback from the one or more first set of users and the one or more second set of users, wherein a false negative indicates a user should have been included in the targeting cluster, and a true negative indicates a user was correctly not included in the targeting cluster;
determining a total number of false positives and a total number of true positives using feedback from the one or more first subgroup of users and the one or more second subgroup of users, wherein a false positive indicates a user should not have been included in the targeting cluster, and a true positive indicates a user was correctly included in the targeting cluster;
determining a precision score using the total number of true positives and the total number of true negatives; and
determining a recall score using the total number of true positives and the total number of false negatives.

5. The method of claim 1 , further comprising:
providing a model evaluation interface to a client device associated with a user of the online system, the model evaluation interface presenting a plurality of targeting models including the first targeting model and the second targeting model, and
wherein retrieving the first targeting model and the second targeting model, is responsive to receiving a selection by the user of the first targeting model and the second targeting model.

6. The method of claim 1 , further comprising:
selecting the first targeting model as a better performing model based at least in part on the scoring;
retrieving an active targeting model, the active targeting model defining how the online system selects targeting attributes for users of the online system;
determining a first test group using the first targeting model, and a second test group using the active targeting model;
determining a holdout group of users of the online system;
selecting a first content item to provide to the first test group and a second content item to provide to the active test group based in part on targeting criteria associated with the content item matching the characteristic of interest; and
providing the first content item to user devices associated with the first test group and the second content item to the second test group.

7. The method of claim 6 , further comprising:
scoring the first targeting model and the active targeting model based in part on conversions of the first content item by users in the first test group and conversions of the second content item by users of the second test group.

8. The method of claim 6 , wherein determining the holdout group further comprises:
determining a random distribution for one or more characteristics; and
selecting users of the online system based in part on the random distribution.

9. The method of claim 1 , further comprising:
selecting the first targeting model as a better performing model based at least in part on the scoring;
partitioning the first group of users into a first test group and a holdout group;
masking a characteristic of users in test group of the first group of users;
identifying users in the test group and in the holdout group, the identification based in part on characteristics associated with the identified users that match targeting criteria of a content item;
providing the content item to the identified users;
determining a number of conversions that occurred for the content item provided to the identified users in the test group and a number of conversions that occurred for the content item by users in the holdout group; and
determining a value of the masked characteristic based in part on a difference between the number of conversions that occurred for the content item provided to the identified users in the test group and the number of conversions that occurred for the content item by users in the holdout group.

10. A method comprising:
predicting, using a first targeting model, a first group of users of an online system as candidates to be in a ta

Assignees

• Facebook Inc
• Facebook Inc

Inventors

• Ye Xingyao
• Martin David Townsend
• Salem Michael Phillip
• Wu Yi
• Xiong Jie
• Xia Mark Teng

Classifications

• G06Q30/0243 Primary

  Comparative campaigns · CPC title

• G06Q30/0255

  based on user history · CPC title

• G06Q30/0277

  Online advertisement · CPC title

• G06Q30/0245

  Surveys · CPC title