Dynamic throughput ingestion of backup sources

Patent metadata

Field	Value
Publication number	US-10298680-B1
Application number	US-201514863178-A
Country	US
Kind code	B1
Filing date	Sep 23, 2015
Priority date	Sep 23, 2015
Publication date	May 21, 2019
Grant date	May 21, 2019

Abstract

A method and apparatus for dynamically adjusting an ingestion rate for backup operations on a source system. The method generally includes monitoring a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload. Based on the monitored resource utilization, the backup system determines a data ingestion rate for backup operations on the source system. The backup system ingests data from the source system to a backup repository at the determined data ingestion rate.

First claim

What is claimed is:

1. A method for dynamically adjusting an ingestion rate for backup operations on a source system, comprising:
monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;
based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;
adjusting the data ingestion rate from the source system, wherein adjusting the data ingestion rate from the source system comprises adjusting an amount of resources allocated to the primary workload at the source system by an equal amount to resources allocated to the backup operations on the source system;
ingesting, at the backup system, data from the source system at the determined data ingestion rate; and
writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

2. The method of claim 1 , further comprising:
comparing the monitored resource utilization to a threshold; and
if the monitored resource utilization exceeds the threshold, decreasing the data ingestion rate such that the backup operations do not interfere with the primary workload.

3. The method of claim 1 , further comprising:
comparing the monitored resource utilization to a threshold; and
if the monitored resource utilization is less than the threshold, increasing the data ingestion rate to accelerate the backup operations.

4. The method of claim 1 , wherein the monitored resource utilization comprises activity at a network interface of the source system.

5. The method of claim 1 , wherein the monitored resource utilization comprises processor idle time at the source system.

6. The method of claim 1 , wherein the monitored resource utilization comprises storage subsystem throughput at the source system.

7. The method of claim 1 , further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to a primary workload at the source system by a greater amount than an adjustment made to resources allocated to backup operations on the source system.

8. The method of claim 1 , further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to backup operations on the source system by a greater amount than an adjustment made to resources allocated to a primary workload at the source system.

9. The method of claim 1 , further comprising comparing the ingested data to data previously stored at the storage repository, wherein the ingested data is written from the backup system to the storage repository based on a comparison of the ingested data to the data previously stored at the storage repository.

10. The method of claim 9 , wherein the backup system is configured to write a pointer to the data previously stored at the storage repository in the event the comparison indicates the ingested data is a same data as the data previously stored at the storage repository.

11. The method of claim 9 , wherein the backup system is configured to write the ingested data to the storage repository in the event the comparison indicates the ingested data is a different data than the data previously stored at the storage repository.

12. A method for dynamically adjusting an ingestion rate for backup operations on a source system, comprising:
monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload, wherein the monitored resource utilization comprises storage system latency at the source system;
based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;
ingesting, at the backup system, data from the source system at the determined data ingestion rate; and
writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

13. A non-transitory computer readable medium comprising instructions that, when executed, perform an operation for dynamically adjusting an ingestion rate for backup operations on a source system, the operations comprising:
monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;
based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;
adjusting the data ingestion rate from the source system, wherein adjusting the data ingestion rate from the source system comprises adjusting an amount of resources allocated to the primary workload at the source system by an equal amount to resources allocated to the backup operations on the source system;
ingesting, at the backup system, data from the source system at the determined data ingestion rate; and
writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

14. The non-transitory computer readable medium of claim 13 , further comprising:
comparing the monitored resource utilization to a threshold; and
if the monitored resource utilization exceeds the threshold, decreasing the data ingestion rate such that the backup operations do not interfere with the primary workload.

15. The non-transitory computer readable medium of claim 13 , further comprising:
comparing the monitored resource utilization to a threshold; and
if the monitored resource utilization is less than the threshold, increasing the data ingestion rate to accelerate the backup operations.

16. The non-transitory computer readable medium of claim 13 , wherein the monitored resource utilization comprises at least one of:
activity at a network interface of the source system;
processor idle time at the source system;
storage subsystem throughput at the source system; or
storage subsystem latency at the source system.

17. A system comprising:
a processor; and
a memory storing one or more instructions, which, when executed by the processor, performs an operation for dynamically adjusting an ingestion rate for backup operations on a source system, the operation comprising:
monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source sy

Assignees

• Cohesity Inc

Inventors

• Thomas Markose
• Manjunath Chinmaya

Classifications

• H04L67/1095 Primary

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• H04L43/16

  Threshold monitoring · CPC title

• H04L43/04

  Processing captured monitoring data, e.g. for logfile generation · CPC title

• H04L43/0888

  Throughput · CPC title