Using propensity score matching to determine metric of interest for unsampled computing devices

Patent metadata

Field	Value
Publication number	US-12436860-B2
Application number	US-202117562875-A
Country	US
Kind code	B2
Filing date	Dec 27, 2021
Priority date	Dec 27, 2021
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Disclosed herein is a system for leveraging telemetry data representing usage of a component installed on a group of sampled computing devices to confidently infer the quality of a user experience and/or the behavior of the component (e.g., an operating system) on a larger group of unsampled computing devices. The system is configured to use a propensity score matching approach to identify a sampled computing device that best represents an unsampled computing device using configuration data that is collected from both the sampled and unsampled computing devices. The quality of the user experience and/or the behavior of the component may be captured by a metric of interest (e.g., a QoS value). Accordingly, the system is configured to use the known metric of interest, determined from the telemetry data collected for the sampled computing device, to determine or predict the metric of interest for the unsampled computing device.

First claim

What is claimed is:

1. A method comprising:
collecting, from each computing device in a set of computing devices, configuration data for an operating system installed on the computing device, wherein:
the set of computing devices comprises a subset of sampled computing devices that have opted in to providing telemetry data indicative of usage of the operating system;
the set of computing devices comprises a subset of unsampled computing devices that have not opted in to providing the telemetry data indicative of the usage of the operating system; and
the configuration data includes attributes that indicate at least a geographic region in which the computing device is located, a version of the operating system installed on the computing device, and a default browser for the computing device;

collecting, from each sampled computing device in the subset of sampled computing devices, the telemetry data indicative of the usage of the operating system, wherein the telemetry data is useable to determine a metric of interest for the sampled computing device;
calculating, by at least one processor and based on a regression model, a propensity score for each computing device in the set of computing devices, wherein the propensity score represents a probability that the computing device is similar to other computing devices based on the configuration data collected from each computing device in the set of computing devices;
for each unsampled computing device in the subset of unsampled computing devices:
using k-Nearest Neighbors (k-NN) to identify, based on the propensity scores calculated for the set of computing devices, a sampled computing device that best represents the unsampled computing device; and
using the propensity score for the unsampled computing device as a multiplier to determine the metric of interest for the unsampled computing device, by applying the multiplier to the metric of interest previously determined for the sampled computing device that best represents the unsampled computing device.

2. The method of claim 1 , wherein the propensity score predicts a label indicative of whether the computing device is one that has opted in to providing the telemetry data, the method further comprising:
comparing the label to a known indicator of whether the computing device has opted in to providing the telemetry data; and
updating the regression model based on the comparing.

3. The method of claim 1 , wherein the attributes further indicate, for a computing device, at least one of: a name for the component, a stock keeping unit (SKU) for the component, a type of install for the component, whether the component was initially installed when the computing device was purchased as new or was upgraded to include the component at a time after the computing device was purchased as new, a build branch for the component, a type of use for the component, whether the component is registered to an account hosted and maintained by a provider of the component, a type of computing device, a manufacturer of the computing device, a number of compute cores in the computing device, a processor manufacturer, a processor model, a total available random access memory (RAM), or a device age.

4. The method of claim 1 , wherein the metric of interest comprises a quality of service (QOS) value.

5. A system comprising:
at least one processor; and
a computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the at least one processor, cause the system to perform operations comprising:
collecting, from each computing device in a set of computing devices, configuration data for a component installed on the computing device, wherein:
the set of computing devices comprises a subset of sampled computing devices that provide telemetry data indicative of usage of the component; and
the set of computing devices comprises a subset of unsampled computing devices that do not provide the telemetry data indicative of the usage of the component;

collecting, from each sampled computing device in the subset of sampled computing devices, the telemetry data indicative of the usage of the component, wherein the telemetry data is useable to determine a metric of interest for the sampled computing device;
calculating, based on a regression model, a propensity score for each computing device in the set of computing devices, wherein the propensity score represents a probability that the computing device is similar to other computing devices based on the configuration data collected from each computing device in the set of computing devices;
for each unsampled computing device in the subset of unsampled computing devices:
using a matching algorithm to identify, based on the propensity scores calculated for the set of computing devices, a sampled computing device that best represents the unsampled computing device; and
using the propensity score for the unsampled computing device in a function to predict the metric of interest for the unsampled computing device, wherein the function uses the metric of interest previously determined for the sampled computing device that best represents the unsampled computing device.

6. The system of claim 5 , wherein the component comprises an operating system or an application.

7. The system of claim 5 , wherein the matching algorithm comprises k-Nearest Neighbors (k-NN), caliper matching, radius matching, or Mahalanobis matching.

8. The system of claim 5 , wherein the propensity score predicts a label indicative of whether the computing device is one that provides the telemetry data, the operations further comprising:
comparing the label to a known indicator of whether the computing device provides the telemetry data; and
updating the regression model based on the comparing.

9. The system of claim 5 , wherein the configuration data includes attributes that indicate, for a computing device, at least a geographic region in which the computing device is located, a version of the component installed on the computing device, and a default browser for the computing device.

10. The system of claim 5 , wherein the configuration data includes attributes that indicate, for a computing device, at least one of: a name for the component, a stock keeping unit (SKU) for the component, a type of install for the component, whether the component was initially installed when the computing device was purchased as new or was upgraded to include the component at a time after the computing device was purchased as new, a build branch for the component, a type of use for the component, whether the component is registered to an account hosted and maintained by a provider of the component, a type of computing device, a manufacturer of the computing device, a number of compute cores in the computing device, a processor manufacturer, a processor model, a total available random access memory (RAM), or a device age.

11. The system of claim 5 , wherein the metric of interest comprises a quality of service (QoS) value.

12. The system of claim 5 , wherein the operations further comprise using the metric of interest predicted for the unsampled computing device to improve the component installed on the unsampled computing device.

13. A method comprising:
collecting, from each computing device in a set of computing devices, configuration data for a component installed on the computing device, wherein:
the set of computing devices comprises a subset of sampled computing devices that provides telemetry data indicative of usage of the component; and
the set of computing devices comprises a subset of unsampled computing devices that do

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Yang Connie
• Millman Eamon Guthrie Cosgrove
• Chai Xiaoyu
• Sadeghi Soheil
• Carter-Thorpe Omari
• Peev Igor Borisov
• Wilssens Steven Marcel Elza
• Mereckis Tomas Aleksas
• Weil Alexander S
• Rowe Stephen C

Classifications

• G06F18/24147

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• G06F18/24137

  Distances to cluster centroïds · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N20/20

  Ensemble learning · CPC title

• G06F11/3452

  Performance evaluation by statistical analysis · CPC title