Advanced retry mechanism for transmitting large datasets

Patent metadata

Field	Value
Publication number	US-10454619-B2
Application number	US-201615346352-A
Country	US
Kind code	B2
Filing date	Nov 8, 2016
Priority date	Nov 8, 2016
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

In non-limiting examples of the present disclosure, systems, methods and devices for transmitting data stored in a source data store to a destination data store are presented. A plurality of data chunks may be received in a first format. At least one upsert operation may be performed on each of the data chunks for converting the first format to a format compatible with a destination data store schema. A transfer of the upserted data chunks to the destination data store may be executed. A determination may be made that at least one of the upserted data chunks has failed to transfer to the destination data store. Failed data chunks may be stored in a failed chunk retry data store. The at least one upsert operation may be re-performed on failed data chunks and a re-execution of the transfer of failed data chunks to the destination data store may be made.

First claim

What is claimed is:

1. A method for transmitting data from a source data store, comprising:
receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;
performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;
executing a first attempt to transfer the modified data chunks to the destination data store;
determining that at least one of the modified data chunks has failed to transfer to the destination data store;
storing the at least one failed data chunk in a failed chunk retry data store;
performing the at least one modification operation on the at least one failed data chunk; and
executing a second attempt to transfer the at least one failed data chunk to the destination data store.

2. The method of claim 1 , further comprising determining a schema of the second format for the destination data store.

3. The method of claim 2 , wherein the at least one modification operation comprises modifying the format of each of the plurality of data chunks to conform to the second format for the destination data store.

4. The method of claim 1 , wherein determining that at least one of the modified data chunks has failed to transfer to the destination data store comprises receiving an indication that a single record contained in the modified data chunks has failed to transfer.

5. The method of claim 1 , wherein determining that at least one of the modified data chunks has failed to transfer to the destination data store comprises receiving an indication that at least one of the modified data chunks was impacted by at least one of: network weather, the destination data store being in a fail state, and the destination data store being in an update state.

6. The method of claim 1 , further comprising:
encrypting, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

7. The method of claim 1 , further comprising:
encrypting, upon performing the at least one modification operation on the at least one failed data chunk, the at least one failed data chunk for transfer to the destination data store.

8. The method of claim 1 , further comprising:
determining whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and
transferring, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.

9. The method of claim 8 , wherein each subsequent attempt to transfer the at least one failed data chunk is linearly scaled based on a linear temporal scale.

10. The method of claim 8 , wherein each subsequent attempt to transfer the at least one failed data chunk is exponentially scaled temporally.

11. The method of claim 8 , wherein a plurality of mapping engines are assigned for transferring the plurality of data chunks to the destination data store, each of the plurality of mapping engines assigned for the transfer of a separate data chunk of the plurality of data chunks.

12. The method of claim 11 , further comprising:
associating, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, metadata with the at least one failed data chunk indicating which of the plurality of mapping engines was assigned to the at least one failed data chunk.

13. The method of claim 1 , wherein the failed chunk retry data store is the source data store.

14. A computer-readable storage device comprising executable instructions that, when executed by a processor, assist with transmitting data from a source data store, the computer-readable storage device including instructions executable by the processor for:
receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;
performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;
executing a first attempt to transfer the modified data chunks to the destination data store;
determining that at least one of the modified data chunks has failed to transfer to the destination data store;
storing the at least one failed data chunk in a failed chunk retry data store;
performing the at least one modification operation on the at least one failed data chunk; and
executing a second attempt to transfer the at least one failed data chunk to the destination data store.

15. The computer-readable storage device of claim 14 , wherein the instructions are further executable by the processor for determining a schema for the second format for the destination data store.

16. The computer-readable storage device of claim 15 , wherein the at least one modification operation comprises modifying the format of each of the plurality of data chunks to conform to the second format for the destination data store.

17. A system for transmitting data from a source data store, comprising:
a memory for storing executable program code; and
a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to:
receive a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;
perform at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;
execute a first attempt to transfer the modified data chunks to the destination data store;
determine that at least one of the modified data chunks has failed to transfer to the destination data store;
store the at least one failed data chunk in a failed chunk retry data store;
perform the at least one modification operation on the at least one failed data chunk; and
execute a second attempt to transfer the at least one failed data chunk to the destination data store.

18. The system of claim 17 , wherein the processor is further responsive to the computer-executable instructions and operative to encrypt, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

19. The system of claim 17 , wherein the processor is further responsive to the computer executable instructions and operative to:
determine whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and
transfer, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.

20. The method of claim 1 , wherein the first format corresponds to a structured format and the second format corresponds to at least one of a non-structured format or a semi-structured format.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Padmanabhan Mohan
• Ray Mayukh
• Kekatpure Aditya

Classifications

• G06F11/14

  Error detection or correction of the data by redundancy in operations (error detection or correction of the data by redundancy in hardware G06F11/16) · CPC title

• G06F3/0644

  Management of space entities, e.g. partitions, extents, pools · CPC title

• G06F3/0661

  Format or protocol conversion arrangements · CPC title

• H04L1/08 Primary

  by repeating transmission, e.g. Verdan system {(H04L1/1858 and H04L1/189 take precedence)} · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title