Balanced control-treatment experiments for software testing

Patent metadata

Field	Value
Publication number	US-2025321727-A1
Application number	US-202218865306-A
Country	US
Kind code	A1
Filing date	May 26, 2022
Priority date	May 26, 2022
Publication date	Oct 16, 2025
Grant date	—

Abstract

A computing system may determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application. The computing system may determine. based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application. a software update for updating the software application to be sent to the computing device. The computing system may send. to the computing device. the software update to trigger the computing device to. in response to receiving the software update. reinstall the control version of the software application at the computing device from a storage device of the computing device.

First claim

1 - 20 . (canceled)

21 . A method comprising:
determining, by one or more processors of a computing system, that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determining, by the one or more processors and based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application, a software update for updating the software application to a treatment version of the software application that is to be sent to the computing device; and sending, by the one or more processors to the computing device, the software update to trigger the computing device to, in response to receiving the software update, reinstall the control version of the software application at the computing device from a storage device of the computing device.

22 . The method of claim 21 , further comprising:
sending, by the one or more processors to a plurality of computing devices in a treatment group associated with the control-treatment experiment for the software application, the software update for updating the software application to the treatment version of the software application.

23 . The method of claim 21 , wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

24 . The method of claim 21 , further comprising:
determining, by the one or more processors, whether the software application installed at a second computing device in the control group is the control version of the software application; and responsive to determining that the software application installed at a second computing device is not the control version of the software application, sending, by the one or more processors to the second computing device, a second software update for updating the software application to the control version of the software application.

25 . The method of claim 21 , further comprising:
determining, by the one or more processors, the control version of the software application as one of: a version of the software application that is fully released to computing devices or a most recent version of the software application prior to the treatment version of the software application.

26 . The method of claim 21 , wherein sending the software update further comprises:
receiving, by the one or more processors and from the computing device, an auto update request associated with the software application; and responsive to receiving the auto update request from the computing device, sending, by the one or more processors to the computing device, the software update to trigger the computing device to, in response to receiving the software update, reinstall the control version of the software application at the computing device from the storage device of the computing device.

27 . The method of claim 21 , wherein sending the software update to the computing device further triggers the computing device to store a copy of the treatment version of the software application in the storage device of the computing device, further comprising:
sending, by the one or more processors to the computing device, an indication that the treatment version of the software application is released as a next official version of the software application that causes the computing device to install the copy of the treatment version of the software application stored at the computing device to replace the control version of the software application.

28 . A computing system comprising:
a memory that stores instructions; and one or more processors that execute the instructions to: determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determine, based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application, a software update for updating the software application to a treatment version of the software application that is to be sent to the computing device; and send, to the computing device, the software update to trigger the computing device to, in response to receiving the software update, reinstall the control version of the software application at the computing device from a storage device of the computing device.

29 . The computing system of claim 28 , wherein the one or more processors are further configured to:
send, to a plurality of computing devices in a treatment group associated with the control-treatment experiment for the software application, the software update for updating the software application to the treatment version of the software application.

30 . The computing system of claim 28 , wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

31 . The computing system of claim 28 , wherein the one or more processors are further configured to:
determine whether the software application installed at a second computing device in the control group is the control version of the software application; and responsive to determining that the software application installed at a second computing device is not the control version of the software application, send, to the second computing device, a second software update for updating the software application to the control version of the software application.

32 . The computing system of claim 28 , wherein the one or more processors are further configured to:
determine the control version of the software application as one of: a version of the software application that is fully released to computing devices or a most recent version of the software application prior to the treatment version of the software application.

33 . The computing system of claim 28 , wherein to send the software update, the one or more processors are further configured to:
receive, from the computing device, an auto update request associated with the software application; and responsive to receiving the auto update request from the computing device, send, to the computing device, the software update to trigger the computing device to, in response to receiving the software update, reinstall the control version of the software application at the computing device from the storage device of the computing device.

34 . The computing system of claim 28 , wherein the software update being sent to the computing device further triggers the computing device to store a copy of the treatment version of the software application in the storage device of the computing device, and wherein the one or more processors are further configured to:
send, to the computing device, an indication that the treatment version of the software application is released as a next official version of the software application that causes the computing device to install the copy of the treatment version of the software application stored at the computing device to replace the control version of

Assignees

• Google Llc

Inventors

• Williams Scott
• Ji Haifeng

Classifications

• G06F11/3612

  by runtime analysis (performance monitoring G06F11/3466) · CPC title

• G06F11/3409

  for performance assessment · CPC title

• G06F8/65 Primary

  Updates (security arrangements therefor G06F21/57) · CPC title

• G06F11/3428 Primary

  Benchmarking · CPC title