Intelligent data backfill and migration operations utilizing event processing architecture

Patent metadata

Field	Value
Publication number	US-10691653-B1
Application number	US-201715695721-A
Country	US
Kind code	B1
Filing date	Sep 5, 2017
Priority date	Sep 5, 2017
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

Disclosed are various embodiments for intelligent backfill and data migration operations performed using an event processing architecture. A backfill system may identify backfill operations to migrate legacy data from a first system to a second system and generate events to provide to an event processor, where each of the events causes a backfill operation to be performed. Access to the events may be selectively controlled using an event processing queue such that the events are processed and the backfill operations are performed when a computing resource has available computing resources, regardless of a time of day.

First claim

Therefore, the following is claimed:

1. A non-transitory computer-readable medium embodying program code executable in at least one computing device that, when executed, causes the at least one computing device to:
identify a plurality of backfill operations to migrate user account data from a first system to a second system;
generate a plurality of events for transmission over a network to an event processor executing in a computing environment, wherein each of the plurality of events comprises a data structure having an operation identifier that, when interpreted, causes at least one of the plurality of backfill operations to be performed in the computing environment;
store the plurality of events in a first-in-first-out event processing queue;
selectively control a rate at which the plurality of events are provided from the first-in-first-out event processing queue to the event processor such that the plurality of events are processed to perform the plurality of backfill operations when an availability of computing resources of the computing environment permits, wherein a receipt of one of the plurality of events spawns a virtual process configured to transform the user account data for a single user account to data capable of interpretation by the second system; and
increase or decrease the rate at which the plurality of events are provided to the event processor based on the availability of the computing resources.

2. The non-transitory computer-readable medium of claim 1 , wherein the event processor is configured to, upon receipt of one of the plurality of events, query a distributed rate limiter system to receive permission to process event data from the first-in-first-out event processing queue.

3. The non-transitory computer-readable medium of claim 2 , wherein the distributed rate limiter system is configured to provide permission to the event processor based at least in part on the availability of the computing resources.

4. A system, comprising:
at least one computing device; and
a memory device embodying program instructions executable in the at least one computing device that, when executed by the at least one computing device, cause the at least one computing device to:
identify a plurality of backfill operations to migrate legacy data from a first system to a second system;
generate a plurality of events to provide to an event processor executing in a computing environment, wherein each of the plurality of events causes at least one of the plurality of backfill operations to be performed; and
selectively provide the plurality of events to the event processor such that the plurality of backfill operations are performed when a computing resource of the computing environment permits, wherein a receipt of one of the plurality of events causes the event processor to spawn a virtual process configured to transform the data to a data format interpretable by the second system.

5. The system of claim 4 , wherein each of the plurality of events comprises a unique identifier corresponding to a type of backfill operation to be performed using event data.

6. The system of claim 4 , wherein the plurality of events are stored in an event processing queue, and the plurality of events are provided to the event processor through the event processing queue.

7. The system of claim 6 , wherein:
the plurality of events are provided to the event processor through the event processing queue in a first-in-first-out (FIFO) manner; and
the event processing queue is configured to remove duplicates of the plurality of events using a unique identifier stored in association with the plurality of events.

8. The system of claim 7 , wherein the event processor is configured to, upon receipt of one of the plurality of events, query a distributed rate limiter system to receive permission to process event data from the event processing queue based at least in part on an availability of computing resources.

9. The system of claim 8 , wherein the distributed rate limiter system is configured to provide permission to the event processor based at least in part on the availability of the computing resources.

10. The system of claim 9 , wherein the plurality of events are selectively provided to the event processor at a predetermined rate of event consumption.

11. The system of claim 10 , wherein the predetermined rate of event consumption or a predetermined rate of generation of the plurality of events is determined based at least in part on an availability of the computing resources.

12. The system of claim 11 , wherein the distributed rate limiter system is configured to:
initially set the predetermined rate of event consumption; and
increase or decrease the predetermined rate of event consumption based at least in part on the availability of the computing resources or the predetermined rate of generation of the plurality of events.

13. The system of claim 12 , wherein the availability of the computing resources is determined by a resource monitor using at least one of: a central processing unit (CPU) utilization, a graphics processing unit (GPU) utilization, available storage space, available memory, and network bandwidth.

14. The system of claim 4 , wherein the virtual process is configured to transform user account data for a single user account to data in the data format interpretable by the second system and maintain a status of a migration of the user account data for the single user account in a data store.

15. A method, comprising:
identifying, by at least one computing device, a plurality of backfill operations to migrate legacy data from a legacy system to a new system;
generating, by the at least one computing device, a plurality of events to provide to an event processor executing in a computing environment, wherein each of the plurality of events causes at least one of the plurality of backfill operations to be performed;
storing, by the at least one computing device, the plurality of events in an event processing queue; and
selectively controlling, by the at least one computing device, access to the plurality of events in the event processing queue for the event processor such that the plurality of events are processed to perform the plurality of backfill operations when a computing resource of the computing environment permits.

16. The method of claim 15 , wherein a receipt of one of the plurality of events causes the event processor to spawn a virtual process configured to transform the data to a data format interpretable by the new system.

17. The method of claim 15 , further comprising querying, by the at least one computing device, a distributed rate limiter system to receive permission to process event data from the event processing queue based at least in part on an availability of the computing resources.

18. The method of claim 15 , wherein access to the plurality of events are selectively controlled based at least in part on a predetermined rate of event consumption determined based at least in part on an availability of the computing resources.

19. The method of claim 18 , further comprising:
setting, by the at least one computing device, the rate of event consumption initially; and
increasing or decreasing, by the at least one computing device, the rate of event consumption based at least in part on the availability of the computing resources.

20. The method of claim 19 , wherein the availability of the computing resources is determined by a resource monitor using at least one of: a central proce

Assignees

• Amazon Tech Inc

Inventors

• Aiken Mark Aran
• Kothandaraman Raghunathan
• Nelson Sam L

Classifications

• G06F16/214 Primary

  Database migration support · CPC title

• G06F21/6218 Primary

  to a system of files or objects, e.g. local or distributed file system or database · CPC title

• G06F21/554

  involving event detection and direct action · CPC title

• G06F9/542

  Event management; Broadcasting; Multicasting; Notifications · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title