Using an rpc framework to facilitate out-of-band data transfers

Patent metadata

Field	Value
Publication number	US-2016112293-A1
Application number	US-201414519888-A
Country	US
Kind code	A1
Filing date	Oct 21, 2014
Priority date	Oct 21, 2014
Publication date	Apr 21, 2016
Grant date	—

Abstract

A system that stores sets of data blocks in extents located in storage devices is described. During operation, a receiving device receives, through an RPC framework, a first call asking to transfer an extent from a sending device to the receiving device. In response, the receiving device opens a port for a data connection that operates outside the RPC framework. The receiving device makes a second call, to the sending device through the RPC framework, asking to stream the extent to the port. The receiving device subsequently receives the extent from the sending device through the port and computes a checksum for the extent. The receiving device also receives a return from the second call, including a checksum for the extent computed by the sending device. If the computed checksum matches the received checksum, the receiving device returns the first call to indicate the transfer operation completed successfully.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
operating a data storage system that stores sets of data blocks in extents that are located in storage devices including a sending device and a receiving device, wherein operating the data storage system includes performing the following operations on the receiving device: receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device; and in response to the first call,
opening a port for a data connection that operates outside the RPC framework,
making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the port,
receiving the extent from the sending device through the port,
computing a checksum for the extent on the receiving device,
receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device, and
if the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call, through the RPC framework, to indicate the transfer operation completed successfully.

2 . The computer-implemented method of claim 1 ,
wherein the data connection is a transmission control protocol (TCP) connection; and wherein the port is a TCP ephemeral port.

3 . The computer-implemented method of claim 1 , wherein the checksum comprises an MD5 hash of the extent.

4 . The computer-implemented method of claim 1 ,
wherein after receiving the extent, the method further comprises storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier; and wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

5 . The computer-implemented method of claim 4 ,
wherein the private identifier is randomly selected from a range of identifiers; and wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

6 . The computer-implemented method of claim 1 , wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

7 . The computer-implemented method of claim 1 , wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

8 . The computer-implemented method of claim 1 , wherein operating the data storage system includes performing the following operations on the sending device:
receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the data connection to the port on the receiving device; and in response to the second call,
streaming the extent through the data connection to the port on the receiving device,
computing a checksum for the extent on the sending device, and
returning the second call along with the computed checksum for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

9 . A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:
operating a data storage system that stores sets of data blocks in extents that are located in storage devices including a sending device and a receiving device, wherein operating the data storage system includes performing the following operations on the receiving device: receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device; and in response to the first call,
opening a port for a data connection that operates outside the RPC framework,
making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the port,
receiving the extent from the sending device through the port,
computing a checksum for the extent on the receiving device,
receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device, and
if the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call, through the RPC framework, to indicate the transfer operation completed successfully.

10 . The non-transitory computer-readable storage medium of claim 9 ,
wherein the data connection is a transmission control protocol (TCP) connection; and wherein the port is a TCP ephemeral port.

11 . The non-transitory computer-readable storage medium of claim 9 , wherein the checksum comprises an MD5 hash of the extent.

12 . The non-transitory computer-readable storage medium of claim 9 ,
wherein after receiving the extent, the method further comprises storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier; and wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

13 . The non-transitory computer-readable storage medium of claim 12 ,
wherein the private identifier is randomly selected from a range of identifiers; and wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

14 . The non-transitory computer-readable storage medium of claim 9 , wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

15 . The non-transitory computer-readable storage medium of claim 9 , wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

16 . The non-transitory computer-readable storage medium of claim 9 , wherein operating the data storage system includes performing the following operations on the sending device:
receiving, through the RPC framework, the request to stream the extent from the sending device through the data connection to the port on the receiving device; and in response to the received request,
streaming the extent through the data connection to the port on the receiving device,
computing a checksum for the extent on the sending device, and
sending the computed checksum for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

17 . A data storage system, comprising:
one or more storage devices including a sending device and a receiving device, wherein the one or more storage devices contain extents that store sets of data blocks; and wherein upon receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device, the receiving

Assignees

• Dropbox Inc

Inventors

• Cowling James
• Turner James

Classifications

• G06F9/547 Primary

  Remote procedure calls [RPC]; Web services · CPC title

• H04L67/02 Primary

  based on web technology, e.g. hypertext transfer protocol [HTTP] · CPC title

• H04L43/10 Primary

  Active monitoring, e.g. heartbeat, ping or trace-route · CPC title

• H04L67/133

  Protocols for remote procedure calls [RPC] · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title