Automatic experience research with a user personalization option method and apparatus

Patent metadata

Field	Value
Publication number	US-12586094-B2
Application number	US-202117542642-A
Country	US
Kind code	B2
Filing date	Dec 6, 2021
Priority date	Dec 6, 2021
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

Techniques for evaluating a user experience experiment designed to use one user experience variant selected from a number of user experience variants as a global-best user experience variant to be used across users relative to a machine model trained to use user data to identify a user-preferred user experience variant. Disclosed systems and methods provide techniques for optimizing user response. In one embodiment, a global-best user experience variant is evaluated by comparing an aggregate user response determined for the global-best user experience variant to an aggregate user response determined using user response predictions determined using the trained machine model, and using the outcome of the comparison to make a recommendation as to which one of the global-best user experience variant and the trained machine model to adopt for providing a user experience to users.

First claim

The invention claimed is:

1 . A method comprising:
receiving, at a computing device, an evaluation request in connection with a user experience experiment designed to adopt a global-best user experience variant, from a number of user experience variants, of software application's user interface, the user experience experiment involving each of a number of user groups being presented with a corresponding experience variant of the software application's user interface; forming, via the computing device, a training user group and an evaluation user group using a pool of users from the number of user groups; obtaining, via the computing device and for each user in the user pool, user data comprising, for each user, an experience variant designation and a corresponding user response metric indicating the user's response, captured via the software application, to a respective user experience variant of the software application's user interface presented, by the software application, to the user, at a computing device of the user, during the user experience experiment, the user data further comprising a number of user attributes; using, via the computing device, the experience variant designation and the corresponding user response metric obtained for each user in the evaluation user group to determine a number of aggregate user responses corresponding to the number of user experience variants; using, via the computing device, the number of aggregate user responses to identify one of the number of user experience variants with a corresponding aggregate user response greater than each other of the number of aggregate user response as the global-best user experience variant for evaluation; generating, via the computing device, training data using the user data corresponding to each user assigned to the training user group; training, via the computing device using a machine learning algorithm, a user response prediction model using the training data; using, via the computing device, the trained user response prediction model to determine, for each user in the evaluation user group, a variant preference prediction identifying one of the number of user experience variants and a corresponding user response prediction; determining, via the computing device, an aggregate user response prediction using the user response prediction determined for each user in the evaluation user group; automatically evaluating, via the computing device, an ability of the global-best user experience variant at optimizing user response relative to the trained user response prediction model's ability using the global-best user experience variant's corresponding aggregate user response and the aggregate user response prediction; identifying, via the computing device, from the number of user experience variants, a user experience variant that, based on the evaluation, optimizes the software application's user response; and presenting, by the software application, the identified user experience variant as its user interface at a number of user computing devices.

2 . The method of claim 1 , identifying a user experience variant further comprising:
using, via the computing device, the trained user response prediction model and a given user's data to identify a specific user experience variant that is used by the software application as its user interface to personalize the software application's user interface for the given user rather than the software application using the global-best user experience variant as its user interface across users if the aggregate user response prediction is greater than the aggregate user response corresponding to the global-best user experience variant of the software application's user interface.

3 . The method of claim 1 , identifying a user experience variant further comprising:
identifying, via the computing device, the global-best user experience variant as the user experience variant that is used by the software application as the user experience variant that is used by the software application as its user interface if the aggregate user response corresponding to the global-best user experience variant of the software application's user interface is greater than the aggregate user response prediction.

4 . The method of claim 1 , further comprising:
selecting, via the computing device, an equal number of users from each user group to form the user pool.

5 . The method of claim 1 , the training data generated for a user assigned to the training user group comprising the number of user attributes and the user's experience variant designation from the user's data, the corresponding experience metric being used as a label for the training data generated for the user.

6 . The method of claim 1 , each aggregate user response, from the number of aggregate user responses, is a metric average corresponding to one of the number of user experience variants, a user experience variant's metric average being determined using each experience metric, from the user data obtained for the evaluation user group, determined to correspond to the user experience variant using the corresponding experience variant designation.

7 . The method of claim 6 , the one of the number of user experience variants identified as the global-best user experience variant for evaluation having a higher metric average than relative to the metric average determined for each other user experience variant.

8 . The method of claim 1 , determining a user response prediction further comprising:
determining a predicted-best user experience variant for each user in the evaluation user group using the trained user response prediction model; determining a user response metric for the predicted-best user experience variant determined for each user.

9 . The method of claim 8 , determining an aggregate user response prediction further comprising:
determining a model average using the user response metric determined for each predicted-best user experience variant.

10 . The method of claim 8 , determining a predicted-best user experience variant for each user in the evaluation user group using the trained user response prediction model further comprising:
determining, for a user in the evaluation user group, a user response prediction for each user experience variant, the determining comprising, for a user experience variant of the number of user experience variants, using the user's attributes and the user experience variant's designation as input to the trained user response prediction model; and selecting, for the user in the evaluation user group, one of the number of user experience variants with a higher user response prediction relative to the user response prediction corresponding to each other user experience variant to be the predicted-best user experience variant.

11 . The method of claim 8 , determining a user response metric for each user's predicted-best user experience, further comprising:
for a user in the evaluation user group, using the corresponding metric from the user's user data as the user's user response metric if the user's experience variant designation from the user's user data designates the predicted-best user experience variant or otherwise using the aggregate user response, from the number of aggregate user responses, corresponding to the user's experience variant determined to be the user's predicted-best user experience variant.

12 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
receiving an evaluation reques

Assignees

• Yahoo Assets Llc

Inventors

• Kotta Guruganesh
• Natkovich Michael
• Jadav Mahendrasinh
• Chen Miao
• Shaw Chandrashekhar
• Kapoor Rahul

Classifications

• G06Q30/0282

  Rating or review of business operators or products · CPC title

• G06N20/00

  Machine learning · CPC title

• G06Q30/0271

  Personalized advertisement · CPC title

• G06Q30/0202

  Market predictions or forecasting for commercial activities · CPC title

• G06Q30/0203 Primary

  Market surveys; Market polls · CPC title