Synchronizing online and offline transfer of data to cloud storage system

Patent metadata

Field	Value
Publication number	US-11301489-B2
Application number	US-201916533586-A
Country	US
Kind code	B2
Filing date	Aug 6, 2019
Priority date	Aug 6, 2019
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

Transportable storage devices are communicatively coupled behind one or more dedicated “edge” data servers to take advantage of the benefits of the data servers and transportable storage devices while limiting their disadvantages. Each edge data server ingests data from one or more client devices and copies the data to one or more of the transportable storage devices for eventual transport to a cloud data center for upload to a cloud storage system. Multiple versions of a segment of data (e.g., of a file or a blob) are maintained at various points in the disclosed system and the disclosed technology ensures that the most recent version is the version ultimately stored on the cloud storage system.

First claim

What is claimed is:

1. A method for use with a cloud storage system, the method comprising:
receiving, at a cloud upload module from a storage device, an object that includes data and metadata, the metadata including an object ID and an ingest timestamp corresponding to a time that the data was ingested into the storage device;
determining whether the object ID exists in an object database of objects stored in the cloud storage system; and
determining a conflict status regarding uploading of the received object into the cloud storage system based on the ingest timestamp of the received object, responsive to the object ID being determined to exist in the object database.

2. The method of claim 1 , wherein the operation of determining a conflict status includes:
ascertaining that an ingest timestamp for the object ID stored in the object database is more recent than the ingest timestamp of the received object;
determining that the conflict status is that a conflict exists between the received object and an existing object in the cloud storage system responsive to the ascertaining; and
disallowing upload of the received object into the cloud storage system responsive to the determined conflict status.

3. The method of claim 2 , wherein the ascertaining further includes:
ascertaining that a last cloud modified timestamp for the object ID stored in the object database is more recent than a last cloud upload timestamp for the object ID stored in the object database.

4. The method of claim 1 , wherein the operation of determining a conflict status includes:
ascertaining that the ingest timestamp of the received object is more recent than an ingest timestamp for the object ID stored in the object database;
determining that the conflict status is that a conflict does not exist between the received object and an existing object in the cloud storage system responsive to the ascertaining; and
allowing upload of the received object into the cloud storage system responsive to the determined conflict status.

5. The method of claim 4 , wherein the ascertaining further includes:
ascertaining that a last cloud upload timestamp for the object ID stored in the object database is more recent than a last cloud modified timestamp for the object ID stored in the object database.

6. The method of claim 4 , wherein the allowing includes:
replacing, with the data in the received object, data in the cloud storage system associated with the object ID; and
replacing, with the metadata in the received object, metadata in the object database associated with the object ID.

7. The method of claim 6 , wherein the replacing the metadata includes:
replacing, with the ingest timestamp of the received object, the ingest timestamp for the object ID stored in the object database; and
replacing, with an entity tag in the received object, an entity tag for the object ID stored in the object database.

8. The method of claim 1 , further including:
allowing upload of at least a portion of the received object into the cloud storage system responsive to the a) object ID being determined to not exist in the object database orb) the object ID being determined to exist in the object database and an entity tag for the object ID stored in the object database matching an entity tag of the received object.

9. The method of claim 1 , wherein the ingest timestamp corresponds to a time that the data was ingested into the storage device remote from the cloud storage system.

10. The method of claim 1 , wherein the ingest timestamp corresponds to a time that the data was ingested into the storage device prior to the receiving operation.

11. One or more tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits of a device a process for use with a cloud storage system, the process comprising:
receiving, at a cloud upload module from a storage device, an object that includes data and metadata, the metadata including an object ID and an ingest timestamp corresponding to a time that the data was ingested into the storage device;
determining whether the object ID exists in an object database of objects stored in the cloud storage system; and
determining a conflict status regarding uploading of the received object into the cloud storage system based on the ingest timestamp of the received object, responsive to the object ID being determined to exist in the object database.

12. The one or more tangible processor-readable storage media of claim 11 , wherein the operation of determining a conflict status includes:
ascertaining that an ingest timestamp for the object ID stored in the object database is more recent than the ingest timestamp of the received object;
determining that the conflict status is that a conflict exists between the received object and an existing object in the cloud storage system responsive to the ascertaining; and
disallowing upload of the received object into the cloud storage system responsive to the determined conflict status.

13. The one or more tangible processor-readable storage media of claim 12 , wherein the ascertaining further includes:
ascertaining that a last cloud modified timestamp for the object ID stored in the object database is more recent than a last cloud upload timestamp for the object ID stored in the object database.

14. The one or more tangible processor-readable storage media of claim 11 , wherein the determining a conflict status includes:
ascertaining that the ingest timestamp of the received object is more recent than an ingest timestamp for the object ID stored in the object database;
determining that the conflict status is that a conflict does not exist between the received object and an existing object in the cloud storage system responsive to the ascertaining; and
allowing upload of the received object into the cloud storage system responsive to the determined conflict status.

15. The one or more tangible processor-readable storage media of claim 14 , wherein the ascertaining further includes:
ascertaining that a last cloud upload timestamp for the object ID stored in the object database is more recent than a last cloud modified timestamp for the object ID stored in the object database.

16. The one or more tangible processor-readable storage media of claim 14 , wherein the allowing upload of the received object into the cloud storage system after the ascertaining includes:
replacing, with the data in the received object, data in the cloud storage system associated with the object ID; and
replacing, with the metadata in the received object, metadata in the object database associated with the object ID.

17. The one or more tangible processor-readable storage media of claim 16 , wherein the replacing the metadata includes:
replacing, with the ingest timestamp of the received object, the ingest timestamp for the object ID stored in the object database; and
replacing, with an entity tag in the received object, an entity tag for the object ID stored in the object database.

18. A system including a processor and memory, the processor configured to execute instructions stored in the memory, comprising:
a storage system storing a plurality of objects, each object including data and metadata, the metadata including an object ID and an ingest timestamp corresponding to a time that the data was ingested into a storage device remote from the storage system; and
an upload engine that manages uploads of objects into the storage system, wherein the upload engine includes:
a r

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Testardi Richard Paul
• D'Amato Andrea
• Joshi Devidas
• Garg Mohit Kumar
• Charles Sebastien
• Gopinadhan Gautam
• Wolfe Stephen Wade
• Renaud John
• Pistor Ernie

Classifications

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title