System and architecture for standardizing and centralizing data movement between systems

Patent metadata

Field	Value
Publication number	US-11836150-B2
Application number	US-202117356113-A
Country	US
Kind code	B2
Filing date	Jun 23, 2021
Priority date	Jun 23, 2021
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A method by one or more computing devices to standardize and centralize data movement between systems. The method includes receiving and storing data source information for a logical data source including authentication and authorization information for accessing a plurality of source systems that store data associated with the logical data source, schema information for a first schema used by a first one of the plurality of source systems to store data associated with the logical data source, and process information for a process including job information for a first job of the process that is for importing data associated with the logical data source from the first source system to a staging area. The method further includes executing the process according to the process information for the process.

First claim

What is claimed is:

1. A method by one or more computing devices implementing a data movement system to standardize and centralize data movement between systems, the method comprising:
receiving and storing data source information for a logical data source including authentication and authorization information for accessing a plurality of source systems that store data associated with the logical data source, wherein the logical data source is a logical representation of data stored across the plurality of source systems representing a same thing but stored using different schemas;
receiving and storing schema information for a first schema used by a first one of the plurality of source systems to store data associated with the logical data source;
receiving and storing schema information for a second schema used by a second one of the plurality of source systems to store data associated with the logical data source;
receiving and storing mapping information for a first mapping between the first schema and a common schema;
receiving and storing mapping information for a second mapping between the second schema and the common schema;
receiving and storing process information for a process including job information for a first job of the process that is for importing data associated with the logical data source from the first source system to a staging area, job information for a second job of the process that is for importing data associated with the logical data source from the second source system to the staging area, and job information for a third job of the process that is for exporting data stored in the staging area to a target system, wherein the process information for the process is received based on a user selection of the logical data source; and
executing the process according to the process information for the process, wherein the execution of the process includes executing the first job of the process by connecting to the first source system using the authentication and authorization information included in the data source information for the logical data source and importing data associated with the logical data source from the first source system to the staging area in a common format using the mapping information for the first mapping, executing the second job of the process by connecting to the second source system using the authentication and authorization information included in the data source information for the logical data source and importing data associated with the logical data source from the second source system to the staging area in the common format using the mapping information for the second mapping, and executing the third job of the process by exporting the data imported from the first source system and the second source system to the target system using mapping information for a third mapping between the common schema and a third schema used by the target system.

2. The method of claim 1 , wherein the mapping information for the first mapping is used to transform the data associated with the logical data source stored in the first source system to the common format and the mapping information for the second mapping is used to transform the data associated with the logical data source stored in the second source system to the common format.

3. The method of claim 1 , wherein the execution of the first job includes validating the data associated with the logical data source stored in the first source system using the schema information for the first schema.

4. The method of claim 1 , wherein the execution of the third job of the process includes connecting to the target system.

5. A non-transitory machine-readable storage medium that provides instructions that, if executed by one or more processors of one or more computing devices implementing a data movement system, are configurable to cause the data movement system to perform operations for standardizing and centralizing data movement between systems, the operations comprising:
receiving and storing data source information for a logical data source including authentication and authorization information for accessing a plurality of source systems that store data associated with the logical data source, wherein the logical data source is a logical representation of data stored across the plurality of source systems representing a same thing but stored using different schemas;
receiving and storing schema information for a first schema used by a first one of the plurality of source systems to store data associated with the logical data source;
receiving and storing schema information for a second schema used by a second one of the plurality of source systems to store data associated with the logical data source;
receiving and storing mapping information for a first mapping between the first schema and a common schema;
receiving and storing mapping information for a second mapping between the second schema and the common schema;
receiving and storing process information for a process including job information for a first job of the process that is for importing data associated with the logical data source from the first source system to a staging area, job information for a second job of the process that is for importing data associated with the logical data source from the second source system to the staging area, and job information for a third job of the process that is for exporting data stored in the staging area to a target system, wherein the process information for the process is received based on a user selection of the logical data source; and
executing the process according to the process information for the process, wherein the execution of the process includes executing the first job of the process by connecting to the first source system using the authentication and authorization information included in the data source information for the logical data source and importing data associated with the logical data source from the first source system to the staging area in a common format using the mapping information for the first mapping, executing the second job of the process by connecting to the second source system using the authentication and authorization information included in the data source information for the logical data source and importing data associated with the logical data source from the second source system to the staging area in the common format using the mapping information for the second mapping, and executing the third job of the process by exporting the data imported from the first source system and the second source system to the target system using mapping information for a third mapping between the common schema and a third schema used by the target system.

6. The non-transitory machine-readable storage medium of claim 5 , wherein the mapping information for the first mapping is used to transform the data associated with the logical data source stored in the first source system to the common format and the mapping information for the second mapping is used to transform the data associated with the logical data source stored in the second source system to the common format.

7. The non-transitory machine-readable storage medium of claim 5 , wherein the execution of the first job includes validating the data associated with the logical data source stored in the first source system using the schema information for the first schema.

8. An apparatus comprising:
one or more processors; and
a non-transitory machine-readable storage medium that provides instructions that, if executed by the one or more processors, are configurable to cause the apparatus to implement a data movement system that performs operations for standardizing and centralizing

Assignees

• Salesforce Inc

Inventors

• Tran Leo Duy
• Angulo David
• Li Edward

Classifications

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F16/212 Primary

  with details for data modelling support · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F21/6218

  to a system of files or objects, e.g. local or distributed file system or database · CPC title