System and method for performing operations on target servers

Patent metadata

Field	Value
Publication number	US-9256506-B1
Application number	US-201313908996-A
Country	US
Kind code	B1
Filing date	Jun 3, 2013
Priority date	Jun 4, 2012
Publication date	Feb 9, 2016
Grant date	Feb 9, 2016

Abstract

A system, computer-readable storage medium storing at least one program, and a computer-implemented method for performing operations on target servers is presented. A request including an operation is received. A set of target servers associated with the operation is identified. The following request processing operations are performed until a predetermined termination condition has been satisfied: a target server in the set of target servers to which the request has not been issued and whose health metrics satisfy health criteria is identified, the request to perform the operation is issued to the target server, and when the request to perform the operation fails at the target server, health metrics for the target server are updated to indicate that the request to perform the operation failed at the target server and health check operation is scheduled to be performed with respect to the target server.

First claim

What is claimed is:

1. A computer-implemented method for performing operations on target servers, performed on a server having at least one processor and memory storing at least one program for execution by the at least one processor to perform the method, comprising:
receiving a request including an operation;
identifying a set of target servers associated with the operation;
performing the following request processing operations until a predetermined termination condition has been satisfied;
identifying a target server in the set of target servers to which the request has not been issued and whose health metrics satisfy health criteria;
issuing the request to perform the operation to the target server;
when the request to perform the operation fails at the target server,
updating health metrics for the target server to indicate that the request to perform the operation failed at the target server; and
scheduling a health check operation to be performed with respect to the target server; and

for each target server with respect to which the health check operation is to be performed:
periodically performing the health check operation with respect to the target server to update health metrics for the target server; and
when the health metrics for the target server satisfy the health criteria, unscheduling the health check operation for the target server.

2. The computer-implemented method of claim 1 , wherein prior to performing the request processing operations, the method includes sorting the target servers in the set of target servers based on network proximity to the server.

3. The computer-implemented method of claim 1 , wherein when the request to perform the operation failed at the target server because the target server was not available, updating the health metrics for the target server includes updating a server availability metric of the target server to indicate that the target server is not available.

4. The computer-implemented method of claim 1 , wherein when the request to perform the operation failed at the target server because the target server issued a pushback signal for the request, updating the health metrics for the target server includes updating a pushback metric of the target server for a quality-of-service class associated with the request to indicate that the target server rejected the request associated with the quality-of-service class.

5. The computer-implemented method of claim 1 , wherein when the request to perform the operation failed at the target server because data associated with the operation to be performed on the target server is not a most-recent version of the data, updating the health metrics for the target server includes updating a replication delay metric of the target server to indicate an estimated amount of time that is required to update the data on the target server to reflect to most-recent version of the data.

6. The computer-implemented method of claim 1 , wherein the predetermined termination condition is satisfied when the request to perform the operation succeeds at one of the target severs in the set of target server.

7. The computer-implemented method of claim 1 , wherein the predetermined termination condition is satisfied when the request to perform the operation failed at all of the target servers in the set of target servers.

8. The computer-implemented method of claim 1 , wherein when the request to perform the operation failed at all of the target servers in the set of target servers, the method includes:
waiting for a predetermined time period to elapse since the request to perform the operation failed at all of the target servers; and
repeating the request processing operations.

9. The computer-implemented method of claim 8 , wherein the predetermined time period includes a time at which at least one scheduled health check operation is performed on at least one target server.

10. The computer-implemented method of claim 1 , wherein performing the health check operation on the target server to update health metrics for the target server includes:
issuing a health check request to the target server;
determining whether the target server is responding to the health check request; and
updating a server availability metric to indicate whether the target server is available or not available.

11. The computer-implemented method of claim 1 , wherein performing the health check operation on the target server to update health metrics for the target server includes:
issuing a health check request to the target server, wherein the health check request is associated with a quality-of-service class;
determining whether the target server sent a pushback signal in response to the health check request; and
updating a pushback metric for the quality-of-service class to indicate whether the target server is rejecting requests associated with the quality-of-service class.

12. The computer-implemented method of claim 1 , wherein performing the health check operation on the target server to update health metrics for the target server includes:
issuing a health check request to the target server to obtain an estimated amount of time that is required to update data on the target server to reflect a most-recent version of the data; and
updating a replication delay metric to indicate the estimated amount of time that is required to update data on the target server to reflect a most-recent version of the data.

13. The computer-implemented method of claim 1 , wherein a respective health metric for a respective target server is
a replication delay metric indicating an estimated amount of time, since a time when a last health check operation was performed with respect to the respective target server, that is required to update data on the respective target server to reflect a most-recent version of the data.

14. The computer-implemented method of claim 1 , wherein respective health metrics for a respective target server is stored on the server.

15. The computer-implemented method of claim 1 , wherein the health check operation is a background operation that is performed independently of the request processing operations.

16. The computer-implemented method of claim 1 , wherein a respective health metric for a respective target server is a respective pushback metric for a respective quality-of-service class indicating whether the respective target server was issuing pushback signals for requests associated with the respective quality-of-service class based on load generated by requests associated with the respective quality-of-service class on the respective server during the last health check operation performed with respect to the respective target server.

17. A system to perform operations on target servers, comprising:
at least one processor;
memory; and
at least one program stored in the memory and executable by the at least one processor, the at least one program comprising instructions to:
receive a request including an operation;
identify a set of target servers associated with the operation;
perform the following request processing operations until a predetermined termination condition has been satisfied:
identify a target server in the set of target servers to which the request has not been issued and whose health metrics satisfy health criteria;
issue the request to perform the operation to the target server;
when the request to perform the operation fails at the target server,
update health metrics for the target server to indicate that the request to perform the operation failed at the ta

Assignees

• Google Inc

Inventors

• Taylor Chris Jorgen
• Ghemawat Sanjay
• Lloyd Alexander
• Fikes Andrew
• Saito Yaz
• Hsieh Wilson Cheng-Yi
• Frost Christopher Cunningham

Classifications

• G06F11/3055 Primary

  Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available (error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction of the data by redundancy in operations G06F11/14; error detection or correction by redundancy in hardware G06F11/16) · CPC title

• G06F11/3089

  Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents (software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466) · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title