Framework for live data migration

Patent metadata

Field	Value
Publication number	US-12450212-B2
Application number	US-202418745838-A
Country	US
Kind code	B2
Filing date	Jun 17, 2024
Priority date	Aug 10, 2021
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

Systems and methods including a framework for migration of live data. The method may comprised, by one or more hardware processors executing program instructions, receiving, at a migration proxy of the framework, code for reading data and writing data compatible with each of a plurality of states of a migration of data in a data store, wherein a service is at least intermittently reading data from and writing data to the data store; determining, by a migration runner of the framework, to perform the migration of the data; initiating, by the migration runner, the migration of the data, wherein the migration comprises a plurality of stages; storing, as the migration progresses through the plurality of stages, and at a migration data store of the framework, a current stage of the migration; and during the migration, using the migration proxy to read data from and write data to the data store.

First claim

What is claimed is:

1. A computer-implemented method for migration of live data, the computer-implemented method comprising, by one or more hardware processors executing program instructions:
receiving code for reading data and writing data compatible with at least a first data format and a second data format different from the first data format;
initiating a migration of data from the first data format to the second data format, wherein the migration comprises a plurality of stages; and
during the migration:
reading data from either the first data format or the second data format;
writing data to both the first and second data formats during stages of the migration in which rollback of the migration is supported; and
writing data to the second data format and not writing data to the first data format during stages of the migration in which rollback of the migration is not supported.

2. The computer-implemented method of claim 1 , wherein the code includes at least: a first read method compatible with the first data format, a second read method compatible with the second data format, a first write method compatible with the first data format, and a second write method compatible with the second data format.

3. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
determining a compatibility of the migration with a service in that accesses the data.

4. The computer-implemented method of claim 3 , wherein determining the compatibility comprises:
determining all nodes of the service;
determining migrations supported by each of the determined nodes; and
determining a maximum migration supported by all live nodes of the determined nodes.

5. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
assigning a monotonically increasing identifier to the migration.

6. The computer-implemented method of claim 1 , wherein the migration is one of a plurality of migrations, and wherein each of the plurality of migrations is assigned a respective one of a plurality of monotonically increasing identifiers.

7. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
determining that the migration is available for the data; and
employing, a first soak time before determining to perform the migration.

8. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
employing a second soak time before finishing the migration.

9. The computer-implemented method of claim 1 , wherein the plurality of stages are ordered.

10. The computer-implemented method of claim 1 , wherein the plurality of stages comprise at least: uninitialized, rolling back, initializing, running, awaiting additional action, awaiting finalization, finishing, and finished.

11. The computer-implemented method of claim 10 , wherein the awaiting additional action stage comprises waiting for completion of an action outside of control of a framework managing the migration.

12. The computer-implemented method of claim 1 , wherein during the migration, a write lock is in place during a change of a current stage of the migration, and a read lock is in place during the current stage of the migration during read or write operations.

13. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
rolling back the migration of the data to an initial state.

14. The computer-implemented method of claim 1 further comprising, by the one or more hardware processors executing program instructions:
providing a user interface configured to receive migration specifications including at least:
migration schema or format changes,
code to migrate data from an old schema or format to a new schema or format, and
code to remove any schema or forma elements no longer needed after the migration.

15. A system comprising:
one or more non-transitory computer-readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the system to:
receive code for reading data and writing data compatible with at least a first data format and a second data format different from the first data format;
initiate a migration of data from the first data format to the second data format, wherein the migration comprises a plurality of stages; and
during the migration:
read data from either the first data format or the second data format;
write data to both the first and second data formats during stages of the migration in which rollback of the migration is supported; and
write data to the second data format and not write data to the first data format during stages of the migration in which rollback of the migration is not supported.

16. A computer program product comprising one or more non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to:
receive code for reading data and writing data compatible with at least a first data format and a second data format different from the first data format;
initiate a migration of data from the first data format to the second data format, wherein the migration comprises a plurality of stages; and
during the migration:
read data from either the first data format or the second data format;
write data to both the first and second data formats during stages of the migration in which rollback of the migration is supported; and
write data to the second data format and not write data to the first data format during stages of the migration in which rollback of the migration is not supported.

Assignees

• Palantir Technologies Inc

Inventors

• Chang Allen
• Male Christopher
• Cohen David
• Ristache Dragos-Florian
• Kramer Danielle
• Garrod John
• Harris Michael
• Zheng Ryan
• Freiberg Stephen

Classifications

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F21/53

  by executing in a restricted environment, e.g. sandbox or secure virtual machine · CPC title

• G06F11/1471

  involving logging of persistent data for recovery · CPC title

• G06F2201/80

  Database-specific techniques · CPC title