Post-experiment network effect estimation based on logged messaging events

Patent metadata

Field	Value
Publication number	US-2020358663-A1
Application number	US-201916405618-A
Country	US
Kind code	A1
Filing date	May 7, 2019
Priority date	May 7, 2019
Publication date	Nov 12, 2020
Grant date	—

Abstract

Computer-implemented techniques for ex post facto accounting for interference from network effects in a one-to-one messaging experiment in an online service. With the techniques, is not necessary to identify isolated, non-interacting communities of users pre-experiment. Instead, unconventionally, a total lift for the treatment feature may be computed post-experiment based on the observed actual messages sent during the experiment by users in the treatment and control groups. Techniques for post-experiment computation of an experiment-specific message response rate, based on observed messages sent, and post-experiment computation of an instant lift, based on overserved message sent, are also disclosed.

First claim

1 . A method performed by a computing system of an online service, the computing system having one or more processors and storage media, the storage media storing one or more computer programs, the one or more computer programs including instructions configured to perform the method and executed by the one or more processors to perform the method, the one or more processors and the storage media provided by one or more computer systems of the computing system, the method comprising:
during an experiment, causing a computer graphical user interface that includes a treatment feature to be displayed at computing devices for a first plurality of user accounts of the online service; during the experiment, causing a computer graphical user interface that includes a control feature, but that does not include the treatment feature, to be displayed at computing devices for a second plurality of user accounts of the online service; wherein, during the experiment, a plurality of messages is sent through the online service; wherein each message, of the plurality of messages, is sent from a respective sender user account to a respective recipient user account; wherein the respective sender user account is a user account of either the first plurality of user accounts or the second plurality of user accounts; wherein the respective recipient user account is a user account of either the first plurality of user accounts or the second plurality of user accounts; wherein the respective recipient user account is a user account other than the respective sender user account; during the experiment, storing in computer storage media a plurality of records for the plurality of messages sent; and based on the plurality of records, determining a first count of messages, of the plurality of messages, that were sent, during the experiment, between user accounts, of the first plurality of user accounts; based on the plurality of records, determining a second count of messages, of the plurality of messages, that were sent, during the experiment, between user accounts, of the second plurality of user accounts; based on the first count of messages and the second count of messages, estimating a total lift for the experiment; and causing a graphical user interface to be displayed that presents the total lift estimated for the experiment.

2 . The method of claim 1 , further comprising:
estimating the total lift for the experiment based on the first count of messages normalized for a ramp percentage and based on the second count of messages normalized for a ramp percentage.

3 . The method of claim 1 , further comprising:
based on the plurality of records, determining a third count of messages, of the plurality of messages, that were sent, during the experiment, from user accounts, of the first plurality of user accounts, to user accounts, of the second plurality of user accounts; based on the plurality of records, determining a fourth count of messages, of the plurality of messages, that were sent, during the experiment, from user accounts, of the second plurality of user accounts, to user accounts, of the first plurality of user accounts; estimating a message response rate for the experiment based on all of: the first count of messages, the second count of messages, the third count of messages, and the fourth count of messages; and causing a graphical user interface to be displayed that presents the message response rate estimated for the experiment.

4 . The method of claim 1 , further comprising:
based on the plurality of records, determining a third count of messages, of the plurality of messages that were sent, during the experiment, from user accounts, of the first plurality of user accounts, to user accounts, of the second plurality of user accounts; based on the plurality of records, determining a fourth count of messages, of the plurality of messages that were sent, during the experiment, from user accounts, of the first plurality of control user accounts, to user accounts, of the second plurality of user accounts; based on the first count of messages, the second count of messages, the third count of messages, and the fourth count of messages, estimating an instant lift for the experiment; and causing a graphical user interface to be displayed that presents the instant lift estimated for the experiment.

5 . The method of claim 1 , further comprising:
during an iteration of a target permutation for variance estimation: assigning a user account a same treatment status at each of a plurality of data processing nodes of a distributed data processing system based on a hash function, an identifier of the user account, and an identifier of the iteration; and wherein the assigning the user account the same treatment status is performed at each of the plurality of data processing nodes without a data processing node of the plurality of data processing nodes communicating over a data communications network with another data processing node of the plurality of data processing node to perform the assigning.

6 . The method of claim 1 , wherein each record of the plurality of records stored during the experiment corresponds to a respective message of the plurality of messages sent; and wherein each record of the plurality of records stored during the experiment contains an identifier of a sending user account of the respective message and contains an identifier of an intended recipient user account of the respective message; and wherein the method further comprises:
after the plurality of records are stored: for each record of the plurality of records, classifying the respective message as treatment-to-treatment, treatment-to-control, control-to-control, or control-to-treatment based on whether the sending user account of the respective message belongs to the first plurality of user accounts or the second plurality of user accounts and based on whether the recipient user account of the respective messages belongs to the first plurality of user accounts or the second plurality of user accounts; and based on the classifying, determining the first count of messages based on a count of messages classified as treatment-to-treatment; and based on the classifying, determining the second count of messages based on a count of messages classified as control-to-control.

7 . The method of claim 1 , wherein a ramp percentage of the experiment is fifty percent.

8 . One or more non-transitory computer-readable media comprising:
one or more computer programs configured for execution by one or more processors and including instructions configured for: during an experiment, causing a computer graphical user interface that includes a treatment feature to be displayed at computing devices for a first plurality of user accounts of an online service; during the experiment, causing a computer graphical user interface that includes a control feature, but that does not include the treatment feature, to be displayed at computing devices for a second plurality of control user accounts of the online service; wherein, during the experiment, a plurality of messages is sent through the online service; wherein each message, of the plurality of messages, is sent from a respective sender user account to a respective recipient user account; wherein the respective sender user account is a user account of either the first plurality of user accounts or the second plurality of user accounts; wherein the respective recipient user account is a user account of either the first plurality of user accounts or the second plurality of user accounts; wherein the respective recipient user account is a user account other than the respective sender user account; during the experiment, storing in computer storage m

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Saint-Jacques Guillaume Benjamin
• Sorenson James Eric
• Chen nanyu
• Xu Ya

Classifications

• H04L51/216

  Handling conversation history, e.g. grouping of messages in sessions or threads · CPC title

• H04L51/52

  for supporting social networking services · CPC title

• H04L51/234

  for tracking messages · CPC title

• H04L51/046 Primary

  Interoperability with other network applications or services · CPC title

• H04L41/22 Primary

  comprising specially adapted graphical user interfaces [GUI] · CPC title