Remote testing

Patent metadata

Field	Value
Publication number	US-9547584-B2
Application number	US-201213413832-A
Country	US
Kind code	B2
Filing date	Mar 7, 2012
Priority date	Mar 8, 2011
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for remote testing. In one aspect, a method includes receiving a first message from a first user device indicating initialization of a client application. The method includes determining that the first user device is a member of a first testing group. The method includes identifying a first testing component associated with the first testing group and capable of altering the client application. The method includes sending the first testing component to the first user device.

First claim

What is claimed is:

1. A method comprising:
receiving, by a server system and from each of a plurality of user devices, a message indicating that a client application installed on the user device is to be executed on the user device, wherein the client application includes a collection of default components that specify a default manner in which the client application is to be executed on the user device;
determining, by the server system and in response to receiving the messages from the plurality of user devices that indicate the client application installed on each of the user devices is to be executed on each of the user devices, that:
(i) a first subset of user devices from the plurality of user devices are assigned to a testing group for an experiment associated with the client application, and
(ii) a second subset of user devices from the plurality of user devices are assigned to a control group for the experiment associated with the client application;
identifying a testing component to send to the user devices in the first subset of user devices based on the first subset of user devices being assigned to the testing group, wherein the testing component is configured to alter the default manner in which the client application is to be executed on the user devices in the first subset of user devices by overriding one or more of the default components in the collection of default components of the client application installed on the user devices in the first subset of user devices;
sending, by the server system, the testing component to the user devices in the first subset of user devices to cause the user devices in the first subset of user devices to execute the client application in the altered manner;
receiving, by the server system and from at least some of the user devices in the first subset of user devices, first data that characterizes user interactions with the client application executing in the altered manner on the at least some of the user devices in the first subset of user devices;
receiving, by the server system and from at least some of the user devices in the second subset of user devices, second data that characterizes user interactions with the client application executing in the default manner on the at least some of the user devices in the second subset of user devices;
determining, by the server system and based on the first data from the at least some of the user devices in the first subset of user devices and the second data from the at least some of the user devices in the second subset of user devices, whether executing the client application in the default manner or executing the client application in the altered manner results in better performance of at least a subset of user devices; and
in response to determining that executing the client application in the altered manner rather than the default manner results in better performance of at least a subset of user devices, taking action by the server system to cause at least one user device that is not among the first subset of user devices to override the one or more default components in the collection of default components of the client application installed on the at least one user device.

2. The method of claim 1 , further comprising:
after sending the testing component to a first user device in the first subset of user devices, removing the first user device from the testing group; and
sending, from the server system and to the first user device as a result of the first user device being removed from the testing group, a second message to the first user device, the second message instructing the first user device to not use the testing component when executing the client application on the first user device.

3. The method of claim 1 , further comprising:
determining, by the server system and in response to receiving the messages from the plurality of user devices, that a third subset of user devices from the plurality of user devices are assigned to a second testing group for the experiment associated with the client application;
identifying a second testing component, different from the testing component to send to the user devices in the third subset of user devices based on the third subset of user devices being assigned to the second testing group, wherein the second testing component is different than the testing component, wherein the second testing component is configured to cause the client application to be executed on the user devices in the third subset of user devices in a second altered manner different from the altered manner, rather than the default manner, by overriding one or more of the default components in the collection of default components of the client application installed on the user devices in the third subset of user devices; and
sending, by the server system, the second testing component to the user devices in the third subset of user devices to cause the user devices in the third subset of user devices to execute the client application in the second altered manner.

4. The method of claim 3 , further comprising:
receiving, by the server system and from at least some of the user devices in the third subset of user devices, third data that characterizes user interactions with the client application executing in the second altered manner on the at least some of the user devices in the third subset of user devices; and
comparing, the first data from the at least some of the user devices in the first subset of user devices with the third data from the at least some of the user devices in the third subset of user devices to determine whether executing the client application in the altered manner or in the second altered manner results in better performance of at least a subset of user devices.

5. The method of claim 1 , wherein sending the testing component to a first user device in the first subset of user devices comprises transmitting the testing component to the first user device before the client application is executed on the first user device to cause the first user device to re-configure the client application with the testing component before the client application is executed on the first user device.

6. The method of claim 1 , wherein sending the testing component to a first user device in the first subset of user devices comprises transmitting the testing component to the first user device while the client application is executing on the first user device so as to cause the first user device to re-configure the client application with the testing component while the client application is executing on the first user device.

7. The method of claim 1 , wherein the testing component comprises modified computer code that is configured to replace default computer code for only a portion of the collection of default components of the client application on the user devices in the first subset of user devices without the client application being re-installed on the user devices in the first subset of user devices.

8. A non-transitory computer storage device encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving, by a server system and from each of a plurality of user devices, a message indicating that a client application installed on the user device is to be executed on the user device, wherein the client application includes a collection of default components that specify a default manner in which the client application is to be executed on the user device;
determining, by the server system and in response to receiving the messages from the plurality of user devices that indicate the client application installed on each of the

Assignees

• Acharya Anish
• Patel Jeson
• Google Inc

Inventors

• Acharya Anish
• Patel Jeson

Classifications

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/25

  Testing of logic operation, e.g. by logic analysers · CPC title

• G06F11/26

  Functional testing · CPC title

• G06F11/22

  Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing · CPC title

• G06Q10/0639

  Performance analysis of employees; Performance analysis of enterprise or organisation operations · CPC title