Meta-data driven ingestion and data processing

Patent metadata

Field	Value
Publication number	US-12346350-B2
Application number	US-202318496543-A
Country	US
Kind code	B2
Filing date	Oct 27, 2023
Priority date	Apr 12, 2023
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

A method implemented by a data processing system for enabling a system to pipeline or otherwise process data in conformance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in conformance with the specified criteria.

First claim

What is claimed is:

1. A method implemented by a data processing system for processing data in conformance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in conformance with the specified criteria, including:
receiving, by a data processing system, a specification that specifies one or more criteria for data by specifying logical metadata and one or more operations to be performed on data associated with the logical metadata;
providing, by a data processing system, a user interface for indicating one or more datasets to be retrieved and processed;
receiving, from the user interface, a user indication of a given dataset;
responsive to at least the user indication, identifying technical metadata for the given dataset;
accessing, from a storage device, a metadata model that specifies relationships among logical metadata and technical metadata;
traversing, by a data processing system, the metadata model accessed from the storage device to identify a relationship among (i) an item of logical metadata in the metadata model corresponding to an item of the logical metadata specified in the specification, and (ii) an item of technical metadata in the metadata model corresponding to an item of the technical metadata for the given dataset;
based on the identified relationship, determining that at least one of the one or more operations are to be performed on data associated with the given dataset and with the item of technical metadata for the given dataset;
generating, by a data processing system, an executable computer program configured to:
perform the determined at least one of the one or more operations on the data associated with the given dataset and with the item of technical metadata for the given dataset such that the data is in accordance with the one or more criteria; and
after performing the determined at least one of the one or more operations, store, in a storage device, the data that is in accordance with the one or more criteria; and

executing, by a data processing system, the executable computer program to perform operations including:
performing, by a data processing system, the determined at least one of the one or more operations on the data associated with the given dataset and with the item of technical metadata for the given dataset such that the data is in accordance with the one or more criteria; and
after performing the determined at least one of the one or more operations, storing, in the storage device, the data that is in accordance with the one or more criteria.

2. The method of claim 1 , wherein the item of the technical metadata for the given dataset is a given item of technical metadata associated with a given data item associated with the given dataset, and wherein the method further includes:
based on the identified relationship, identifying that the given item of data is associated with the given item of technical metadata corresponding to the item of technical metadata in the metadata model; and
identifying that the one or more criteria are applicable to the given item of data.

3. The method of claim 1 , further including:
based on the traversing, identifying a data quality control associated with the item of technical metadata in the metadata model corresponding to an item of technical metadata for the given dataset.

4. The method of claim 1 , further including:
updating the metadata model based on execution of the executable computer program;
detecting that the execution causes updating of the metadata model;
traversing the metadata model to identify one or more relationships among (i) data added to the metadata model based on the execution, and (ii) other data in the metadata model; and
based on the identified one or more relationships, updating the executable computer program in accordance with the one or more relationships among (i) the data added to the metadata model based on the updating, and (ii) the other data in the metadata model.

5. The method of claim 4 , further including:
based on determining no additional updates to the metadata model,
outputting the executable computer program with updated instructions for execution; or
storing the executable computer program for execution.

6. The method of claim 1 , further including:
receiving, from a metadata system, identifiers of data that are candidates for processing in accordance with the one or more criteria; and
causing the user interface to render graphical visualizations of the identifiers.

7. The method of claim 1 , further including:
accessing the given dataset.

8. One or more non-transitory machine-readable hardware storage devices for processing data in conformance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in conformance with the specified criteria, the one or more machine-readable hardware storage devices storing instructions that are executable by one or more processing devices to perform operations including:
receiving a specification that specifies one or more criteria for data by specifying logical metadata and one or more operations to be performed on data associated with the logical metadata;
providing a user interface for indicating one or more datasets to be retrieved and processed;
receiving, from the user interface, a user indication of a given dataset;
responsive to at least the user indication, identifying technical metadata for the given dataset;
accessing, from a storage device, a metadata model that specifies relationships among logical metadata and technical metadata;
traversing the metadata model accessed from the storage device to identify a relationship among (i) an item of logical metadata in the metadata model corresponding to an item of the logical metadata specified in the specification, and (ii) an item of technical metadata in the metadata model corresponding to an item of technical metadata for the given dataset;
based on the identified relationship, determining that at least one of the one or more operations are to be performed on data associated with the given dataset and with the item of technical metadata for the given dataset;
generating an executable computer program configured to:
perform the determined at least one of the one or more operations on the data associated with the given dataset and with the item of technical metadata for the given dataset such that the data is in accordance with the one or more criteria; and
after performing the determined at least one of the one or more operations, store, in a storage device, the data that is in accordance with the one or more criteria; and

executing the executable computer program to perform operations including:
performing the determined at least one of the one or more operations on the data associated with the given dataset and with the item of technical metadata for the given dataset such that the data is in accordance with the one or more criteria; and
after performing the determined at least one of the one or more operations, storing, in the storage device, the data that is in accordance with the one or more criteria.

9. The one or more non-transitory machine-readable hardware storage devices of claim 8 , wherein the item of the technical metadata for the given dataset is a given item of technical metadata associated with a given data item associated with the given dataset, and wherein the operations further include:
based on the identified relationship, identifying that the g

Assignees

• Ab Initio Technology Llc

Inventors

• Isman Marshall A
• Weiss Adam
• Egenolf Jonah
• Parks Robert
• Maclean John
• Mellon Richard
• Radivojevic Dusan
• Veiser Paul
• Khader Mazin Woodrow

Classifications

• G06F16/211

  Schema design and management · CPC title

• G06F16/908 Primary

  using metadata automatically derived from the content · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F9/4494

  data driven · CPC title

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title