Scalable metadata-driven data ingestion pipeline
US-2023259518-A1 · Aug 17, 2023 · US
US12153971B2 · US · B2
| Field | Value |
|---|---|
| Publication number | US-12153971-B2 |
| Application number | US-202318243609-A |
| Country | US |
| Kind code | B2 |
| Filing date | Sep 7, 2023 |
| Priority date | Aug 17, 2022 |
| Publication date | Nov 26, 2024 |
| Grant date | Nov 26, 2024 |
A practical reading order for non-experts. Skip the full description unless you need deep technical detail.
What the patent document calls the invention.
A short plain-language summary of the technical disclosure.
Who owns or filed the patent and who is credited as inventor.
Filing, priority, publication, and grant dates set the timeline.
The legal scope of protection — read this for what is actually claimed.
Technology tags used to group this patent with similar filings.
Prior art links and similar publications in this corpus.
Official abstract text for this publication.
Disclosed is an execution information sharing system that duplicates execution information to a provider target (and other targets) as it is being loaded to a consumer target. A first log information object and a second log information object are generated. The first and second log information objects comprise information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written. A first event unloader and a second event unloader are generated based on the first and second log information objects respectively, wherein the first and second event unloaders are both linked to the application using a mapping. In response to receiving execution information from the application, the execution information is forwarded to the consumer target and the provider target using the first event unloader and the second event unloader respectively.
Opening claim text (preview).
What is claimed is: 1. A method comprising: generating a first log information object and a second log information object, the first and second log information objects comprising information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written; generating a first event unloader and a second event unloader based on the first and second log information objects respectively, wherein the first event unloader and the second event unloader are both linked to the application using a mapping; and in response to receiving execution information from the application, forwarding, by a processing device, the execution information to the consumer target and the provider target using the first event unloader and the second event unloader respectively. 2. The method of claim 1 , further comprising: associating the first event unloader with a log correlation ID of the application; updating the mapping to indicate that the first event unloader is linked to the log correlation ID of the application; associating the second event unloader with the log correlation ID of the application; and updating the mapping to indicate that the second event unloader is linked to the log correlation ID of the application. 3. The method of claim 2 , further comprising: identifying the log correlation ID of the application in response to receiving the execution information; identifying the first event unloader and the second event unloader as being associated with the log correlation ID of the application based on the mapping; and retrieving the first event unloader and the second event unloader. 4. The method of claim 1 , wherein forwarding the execution information to the consumer target comprises: determining a log level of the application and a trace level of the application set by the provider account; filtering from the execution information, execution logs having a higher log level than the log level set by the provider account; filtering from the execution information, trace events having a higher trace level than the trace level set by the provider account; and forwarding a remainder of the execution information to the consumer target. 5. The method of claim 1 , further comprising: specifying, by the provider account, a log level of the application by selecting from a set of predefined log level values; and specifying, by the provider account, a trace level of the application by selecting from a set of predefined trace level values. 6. The method of claim 1 , wherein generating the second log information object comprises: obtaining share information from an imported database from which the application is accessed; extracting from the share information, provider account information; and generating the second log object based on the provider account information. 7. The method of claim 6 , wherein the provider account information comprises: a provider task pipe ID; and a provider staging file name. 8. The method of claim 1 , wherein the consumer target and the provider target each correspond to an event table. 9. A system comprising: a memory; and a processing device operatively coupled to the memory, the processing device to: generate a first log information object and a second log information object, the first and second log information objects comprising information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written; generate a first event unloader and a second event unloader based on the first and second log information objects respectively, wherein the first event unloader and the second event unloader are both linked to the application using a mapping; and in response to receiving execution information from the application, forward the execution information to the consumer target and the provider target using the first event unloader and the second event unloader respectively. 10. The system of claim 9 , wherein the processing device is further to: associate the first event unloader with a log correlation ID of the application; updating the mapping to indicate that the first event unloader is linked to the log correlation ID of the application; associate the second event unloader with the log correlation ID of the application; and update the mapping to indicate that the second event unloader is linked to the log correlation ID of the application. 11. The system of claim 10 , wherein the processing device is further to: identify the log correlation ID of the application in response to receiving the execution information; identify the first event unloader and the second event unloader as being associated with the log correlation ID of the application based on the mapping; and retrieve the first event unloader and the second event unloader. 12. The system of claim 9 wherein to forward the execution information to the consumer target, the processing device is to: determine a log level of the application and a trace level of the application set by the provider account; filter from the execution information, execution logs having a higher log level than the log level set by the provider account; filter from the execution information, trace events having a higher trace level than the trace level set by the provider account; and forward a remainder of the execution information to the consumer target. 13. The system of claim 9 , wherein the processing device is further to: specify, by the provider account, a log level of the application by selecting from a set of predefined log level values; and specify, by the provider account, a trace level of the application by selecting from a set of predefined trace level values. 14. The system of claim 9 , wherein to generate the second log information object, the processing device is to: obtain share information from an imported database from which the application is accessed; extract from the share information, provider account information; and generate the second log object based on the provider account information. 15. The system of claim 14 , wherein the provider account information comprises: a provider task pipe ID; and a provider staging file name. 16. The system of claim 8 , wherein the consumer target and the provider target each correspond to an event table. 17. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device, cause the processing device to: generate a first log information object and a second log information object, the first and second log information objects comprising information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written; generate a first event unloader and a second event unloader based on the first and second log information objects respectively, wherein the first event unloader and the second event unloader are both linked to the application using a mapping; and in response to receiving execution information from the application, forward, by the processing device, the execution information to the consumer target and the provider target using the first event unloader and the second event unloader respectively. 18. The non-transit
Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title
Grid computing · CPC title
Event management; Broadcasting; Multicasting; Notifications · CPC title
Related publications grouped by family.
Answers are generated from the same data shown on this page.