Automatic experience research with a user personalization option method and apparatus

Patent metadata

Field	Value
Publication number	US-2023177539-A1
Application number	US-202117542642-A
Country	US
Kind code	A1
Filing date	Dec 6, 2021
Priority date	Dec 6, 2021
Publication date	Jun 8, 2023
Grant date	—

Abstract

Techniques for evaluating a user experience experiment designed to use one user experience variant selected from a number of user experience variants as a global-best user experience variant to be used across users relative to a machine model trained to use user data to identify a user-preferred user experience variant. Disclosed systems and methods provide techniques for optimizing user response. In one embodiment, a global-best user experience variant is evaluated by comparing an aggregate user response determined for the global-best user experience variant to an aggregate user response determined using user response predictions determined using the trained machine model, and using the outcome of the comparison to make a recommendation as to which one of the global-best user experience variant and the trained machine model to adopt for providing a user experience to users.

First claim

1 . A method comprising:
receiving, at a computing device, an evaluation request in connection with a user experience experiment designed to adopt a global-best user experience variant, from a number of user experience variants, for use across users, the user experience experiment involving a number of user groups corresponding to the number of user experience variants; forming, via the computing device, a training user group and an evaluation user group using a pool of users from the number of user groups; obtaining, via the computing device and for each user in the user pool, user data comprising, for each user, an experience variant designation and a corresponding user response metric from the user experience experiment and a number of user attributes; using, via the computing device, the experience variant designation and the corresponding user response metric obtained for each user in the evaluation user group to determine a number of aggregate user responses corresponding to the number of user experience variants; using, via the computing device, the number of aggregate user responses to identify one of the number of user experience variants with a corresponding aggregate user response greater than each other of the number of aggregate user response as the global-best user experience variant for evaluation; generating, via the computing device, training data using the user data corresponding to each user assigned to the training user group; training, via the computing device using a machine learning algorithm, a user response prediction model using the training data; using, via the computing device, the trained user response prediction model to determine, for each user in the evaluation user group, a variant preference prediction identifying one of the number of user experience variants and a corresponding user response prediction; determining, via the computing device, an aggregate user response prediction using the user response prediction determined for each user in the evaluation user group; automatically evaluating, via the computing device, an ability of the global-best user experience variant at optimizing user response relative to the trained user response prediction model's ability using the global-best user experience variant's corresponding aggregate user response and the aggregate user response prediction; and making, via the computing device, a recommendation for providing a user experience to users based on the evaluation.

2 . The method of claim 1 , making a recommendation further comprising:
automatically making, via the computing device, a recommendation to use the trained user response prediction model to personalize a user experience using the user's data rather than using the global-best user experience variant across users if the aggregate user response prediction is greater than the global-best user experience variant's corresponding aggregate user response.

3 . The method of claim 1 , further comprising:
automatically making, via the computing device, a recommendation to use the global-best user experience variant rather than the trained user response prediction model if the global-best user experience variant's corresponding aggregate user response is greater than the aggregate user response prediction.

4 . The method of claim 1 , further comprising:
selecting, via the computing device, an equal number of users from each user group to form the user pool.

5 . The method of claim 1 , the training data generated for a user assigned to the training user group comprising the number of user attributes and the user experience variant designation, the corresponding experience metric being used a label for training data generated for the user.

6 . The method of claim 1 , each aggregate user response, from the number of aggregate user responses, is a metric average corresponding to one of the number of user experience variant, a user experience variant's metric average being determined using each experience metric, from the user data obtained for the evaluation user group, determined to correspond to the user experience variant using the corresponding experience variant designation.

7 . The method of claim 6 , the one of the number of user experience variants identified as the global-best user experience variant for evaluation having a higher metric average than relative to the metric average determined for each other user experience variant.

8 . The method of claim 1 , determining a user response prediction further comprising:
determining a predicted-best user experience variant for each user in the evaluation user group using the trained user response prediction model; determining a user response metric for the predicted-best user experience variant determined for each user.

9 . The method of claim 8 , determining an aggregate user response prediction further comprising:
determining a model average using the user response metric determined for each predicted-best user experience variant.

10 . The method of claim 8 , determining a predicted-best user experience variant for each user in the evaluation user group using the trained user response prediction model further comprising:
determining, for a user in the evaluation user group, a user response prediction for each user experience variant, the determining comprising, for a user experience variant of the number of user experience variants, using the user's attributes and the user experience variant's designation as input to the trained user response prediction model; and selecting, for the user in the evaluation user group, one of the number of user experience variants with a higher user response prediction relative to the user response prediction corresponding to each other user experience variant to be the predicted-best user experience variant.

11 . The method of claim 8 , determining a user response metric for each user's predicted-best user experience, further comprising.
for a user in the evaluation user group, using the corresponding metric from the user's user data as the user's user response metric if the user experience variant designation from the user's user data designates the predicted-best user experience variant or otherwise using the aggregate user response, from the number of aggregate user responses, corresponding to the user experience variant determined to be the user's predicted-best user experience variant.

12 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
receiving an evaluation request in connection with a user experience experiment designed to adopt a global-best user experience variant, from a number of user experience variants, for use across users, the user experience experiment involving a number of user groups corresponding to the number of user experience variants; forming a training user group and an evaluation user group using a pool of users from the number of user groups; obtaining, for each user in the user pool, user data comprising, for each user, an experience variant designation and a corresponding user response metric from the user experience experiment and a number of user attributes; using the experience variant designation and the corresponding user response metric obtained for each user in the evaluation user group to determine a number of aggregate user responses corresponding to the number of user experience variants; using the number of aggregate user responses to identify one of the number of user experience variants with a corresponding aggregate user response greater

Assignees

• Yahoo Assets Llc

Inventors

• Kotta Guruganesh
• Natkovich Michael
• Jadav Mahendrasinh
• Chen Miao
• Shaw Chandrashekhar
• Kapoor Rahul

Classifications

• G06Q30/0203 Primary

  Market surveys; Market polls · CPC title

• G06N20/00

  Machine learning · CPC title

• G06Q30/0282

  Rating or review of business operators or products · CPC title

• G06Q30/0202

  Market predictions or forecasting for commercial activities · CPC title

• G06Q30/0271

  Personalized advertisement · CPC title