Medical ETL task dispatching method, system and apparatus based on multiple centers

Patent metadata

Field	Value
Publication number	US-12119108-B2
Application number	US-202318363701-A
Country	US
Kind code	B2
Filing date	Aug 1, 2023
Priority date	Aug 31, 2022
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

The present disclosure discloses a medical ETL task dispatching method, system and apparatus based on multiple centers. The method includes following steps: step S1: testing and verifying ETL tasks; step S2: deploying the ETL tasks to a hospital center, and dispatching the ETL tasks to a plurality of executors for execution; step S3: screening an executor set meeting resource demands of ETL tasks to be dispatched; step S4: calculating a current task load of each executor in the executor set; step S5: selecting the executor with a minimum current task load to execute the ETL tasks; and step S6: selecting, by the dispatching machine, the ETL tasks from executor active queues according to a priority for execution. The present disclosure selects the most suitable executor by analyzing a serving index as a task to be dispatched on a current dispatching machine.

First claim

What is claimed is:

1. A medical ETL task dispatching method based on multiple centers, comprising steps of:
step S 1 : generating ETL tasks, collecting resource demands of the ETL tasks and determining a time prediction equation by using a test machine, and testing and verifying the ETL tasks;
step S 11 : generating the ETL tasks, operating the ETL tasks through the test machine, dividing data in an ETL task operating process into test data and verification data, and collecting resource demands of the test data and resource demands of the verification data, respectively;
step S 12 : reading a data volume and a data reading rate of the test data by using the test machine, and determining the time prediction equation PRE_T i k according to the data volume and the data reading rate;

PRE_T
i
k

=

a
⁢

V
k

I
k

+
b

wherein V k represents a data volume, to be processed, of an ETL task Task i in a stage k, I k represents the data reading rate, and a and b are constant indexes;
step S 13 : obtaining prediction time of the ETL tasks corresponding to the test data by using the time prediction equation;
step S 14 : verifying the resource demands and the prediction time, and when the resource demands of the test data meet the resource demands of the verification data, and meanwhile, a difference value between the prediction time and actual execution time of the ETL tasks corresponding to the verification data is less than a preset threshold value, completing test and verification of the ETL tasks;
step S 2 : deploying the ETL tasks to a hospital center, and dispatching, by the hospital center, the ETL tasks to a plurality of executors through a dispatching machine for execution;
step S 21 : deploying the ETL tasks to the hospital center;
step S 22 : determining the prediction time of the ETL tasks by using the time prediction equation, wherein step S 22 comprises determining the prediction time of the ETL tasks, by using the time prediction equation, through the number of ETL tasks to be processed in a current stage and a data reading rate of the hospital center;
step S 23 : determining a priority of the ETL tasks by using the prediction time, wherein a shortest task priority principle is used to determine the priority, and the shortest task priority principle stipulates that a ETL task with shorter remaining processing time in the current stage has a higher priority to reduce average waiting time of all ETL tasks;
step S 24 : dispatching, by the dispatching machine, the ETL tasks to the executors for execution according to the priority of the ETL tasks;
step S 241 : initiating, by the dispatching machine, active task queues and expired task queues;
step S 242 : adding the ETL tasks to the active task queues according to the priority;
step S 243 : when the ETL tasks in the active task queues are empty, exchanging the active task queues and the expired task queues, and continuously performing, by the dispatching machine, distributing and dispatching from the active task queues;
step S 3 : collecting and counting up, by the dispatching machine, resource index vectors reported by each executor and resource demand vectors of ETL tasks to be dispatched in the current stage, and screening an executor set meeting resource demands of the ETL tasks to be dispatched;
step S 4 : calculating a current task load of each executor in the executor set;
step S 5 : selecting, by the dispatching machine, the executor with a minimum current task load to execute the ETL tasks according to the current task load of each executor; and
step S 6 : adding, by the dispatching machine, the ETL tasks to executor active queues, determining the priority of the ETL tasks in the executor active queues according to prediction time determined by the prediction equation, and selecting, by the dispatching machine, the ETL tasks from the executor active queues according to the priority for execution;
in the ETL task execution process, setting an ETL task operation time threshold value, when the ETL task execution time is greater than or equal to the ETL task operation time threshold value, pausing execution of the ETL tasks, adding the ETL tasks to the executor expire queues, and waiting for next-time dispatching;
in the ETL task execution process, detecting ETL task stage information, when stages are switched, pausing execution of the ETL tasks, adding the ETL tasks to dispatching machine expired task queues, and waiting for re-dispatching by the dispatching machine; and
in the ETL task execution process, when the executor active queues are empty after executor dispatching, exchanging the executor active queues and the executor expire queues, and continuously performing, by the dispatching machine, dispatching and execution from the executor active queues.

2. The medical ETL task dispatching method based on multiple centers according to claim 1 , wherein step S 3 comprises sub-steps of:
step S 31 : collecting and counting up, by the dispatching machine, resource index vectors of any executor;
step S 32 : collecting and counting up, by the dispatching machine, the resource demand vectors of the ETL tasks to be dispatched in the current stage; and
step S 33 : screening the executor set meeting the resource demands of the ETL tasks to be dispatched by using the resource index vectors and the resource demand vectors.

3. The medical ETL task dispatching method based on multiple centers according to claim 1 , wherein step S 4 comprises sub-steps of:
step S 41 : calculating a sum of prediction time of all the ETL tasks in each executor active queue and each executor expire queue in each executor set by using the time prediction equation; and
step S 42 : calculating current task loads corresponding to the executors through the sum of the prediction time and a collection of all the ETL tasks.

4. The medical ETL task dispatching method based on multiple centers according to claim 1 , wherein when the current task loads of the plurality of executors are the same in step S 5 , the executor with a minimum value is screened out to perform dispatching and execution on the ETL tasks according to the resource index vectors of the executors in the current stage and the resource demand vectors of the ETL tasks in the current stage in combination with resource weight values of the executors.

5. The medical ETL task dispatching method based on multiple centers according to claim 4 , wherein when the plurality of executors are screened out in step S 5 , one executor is randomly selected to perform dispatching and execution on the ETL tasks.

6. A system for implementing the medical ETL task dispatching method based on multiple centers according to claim 1 , comprising:
a test module, configured to collect ETL task operation data and determine resource demands of ETL tasks and a time prediction equation;
a hospital center module, configured to deploy the ETL tasks and submit the ETL tasks to a dispatching machine module to perform dispatching and distributing of the ETL tasks;
the dispatching machine module, configured to calculate executor sources a

Assignees

• Zhejiang Lab

Inventors

• Li Jingsong
• Xiang Wenchao
• Deng Guangyuan
• Zhou Tianshu
• Tian Yu

Classifications

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G16H10/60

  for patient-specific data, e.g. for electronic patient records · CPC title

• G06F9/505

  considering the load · CPC title

• G16H40/20 Primary

  for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms · CPC title