Declarative external data source importation, exportation, and metadata reflection utilizing http and HDFS protocols

Patent metadata

Field	Value
Publication number	US-10976907-B2
Application number	US-201514864520-A
Country	US
Kind code	B2
Filing date	Sep 24, 2015
Priority date	Sep 26, 2014
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

Techniques are disclosure for a data enrichment system that enables declarative external data source importation and exportation. A user can specify via a user interface input for identifying different data sources from which to obtain input data. The data enrichment system is configured to import and export various types of sources storing resources such as URL-based resources and HDFS-based resources for high-speed bi-directional metadata and data interchange. Connection metadata (e.g., credentials, access paths, etc.) can be managed by the data enrichment system in a declarative format for managing and visualizing the connection metadata.

First claim

What is claimed is:

1. A method comprising:
receiving, by a computing system of a data enrichment system, a request to access a data source from the data enrichment system,
wherein the data enrichment system is a big data system that is configured to transform, repair and/or enrich a data set, and provide a user with visual recommendations on an interactive graphical user interface for transformation, repairing and/or enriching the data set,
wherein the request includes identification information of the data source and a source type of the data source;
determining, by the computing system, the source type of the data source based on the received request;
determining, by the computing system, using the identification information of the data source, that the data enrichment system does not have a connection to the data source;
in response to determining that the data enrichment system does not have the connection to the data source:
rendering, by the computing system, the interactive graphical user interface to receive connection information for the data source, the connection information based on the source type of the data source;
receiving, by the computing system, via the interactive graphical user interface, a set of connection parameters for the data source, the set of connection parameters including a connection name, a connection type, a data source location, and credential information;
storing the set of connection parameters in a data repository system accessible to the data enrichment system;
establishing, by the computing system, using the set of connection parameters for the data source, a connection between the data enrichment system and the data source in order for the user to interactively manage data accessed from the data source; and
generating, by the computing system, a profile for the data set accessed from the data source via the connection established between the data enrichment system and the data source,

automatically generating, independently by the data enrichment system, a data service corresponding to the request;
determining, by the data service, the profile for the data set; and
automatically generating, independently by the data service, a transformed data set using the profile determined by the data service.

2. The method of claim 1 , further comprising:
storing, using a data structure in the data repository system, the set of connection parameters in association with the data service,
wherein the transformed data set is generated based on the data set accessed from the data source.

3. The method of claim 1 , wherein the source type includes a cloud-based storage system, a distributed storage system, a web service, or a uniform resource locator (URL).

4. The method of claim 1 , further comprising:
querying, using the identification information, the data repository system to determine whether the data enrichment system has a connection to the data source;
upon determining that the data enrichment system has a connection to the data source, accessing from the data repository system, connection information for the connection; and
rendering the interactive graphical user interface to indicate the connection information for the connection to the data source.

5. The method of claim 1 , wherein the credential information includes an access key and a password defined for access to the data source.

6. The method of claim 1 , further comprising:
accessing, using the identification information, the data repository system to determine that the data enrichment system has a plurality of connections to the data source;
rendering a user interface that identifies each of the plurality of connections to the data source; and
receiving, via the user interface that identifies each of the plurality of connections, input indicating a selection of one of the plurality of connections.

7. The method of claim 1 , wherein storing the set of connection parameters and the credential information includes:
generating one or more data structures defining the data source, the one or more data structures including the set of connection parameters and the credential information, wherein the one or more data structures are stored in the data repository system.

8. The method according to claim 1 , wherein the data source comprises one of a public cloud storage service, a private cloud storage service, and a web-based data source.

9. A data enrichment system comprising:
a plurality of data sources;
a data repository system; and
a cloud computing infrastructure system comprising:
one or more processors communicatively coupled to the plurality of data sources over at least one communication network; and
a memory coupled to the one or more processors, the memory storing instructions to provide a data enrichment service, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive a request to access a data source of the plurality of data sources, wherein the request includes identification information of the data source and a source type of the data source;
determine, using the identification information of the data source, that the data enrichment system does not have a connection to the data source,

wherein the data enrichment system is a big data system that is configured to transform, repair and/or enrich a data set, and provide a user with visual recommendations on an interactive graphical user interface for transformation, repairing and/or enriching the data set;
determine the source type of the data source based on the received request;
in response to determining the source type of the data source and in response to determining that the data enrichment system does not have the connection to the data source:
render the interactive graphical user interface to receive connection information for the data source, the connection information based on the source type of the data source;
receive, via the interactive graphical user interface, a set of connection parameters, the set of connection parameters for the data source including a connection name, a connection type, a data source location, and credential information;
store the set of connection parameters in the data repository system accessible to the data enrichment system;
establish, using the set of connection parameters, a connection between the data enrichment system and the data source in order for the user to interactively manage data accessed from the data source; and
generate a profile for the data set accessed from the data source via the connection established between the data enrichment system and the data source

automatically generate, independently by the data enrichment system, a data service corresponding to the request;
determine, by the data service, the profile for the data set; and
automatically generate, independently by the data service, a transformed data set using the profile determined by the data service.

10. The data enrichment system of claim 9 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
store, using a data structure in the data repository system, the set of connection parameters in association with the data service,
wherein the transformed data set is generated based on the data set accessed from the data source.

11. The data enrichment system of claim 9 , wherein the source type includes a cloud-based storage system, a distributed storage system, a web service, or a uniform resource locator (URL).

12. The data enrichment system of claim 9 , wherein the instructions, when executed by the one or more processors, fu

Assignees

• Oracle Int Corp

Inventors

• Stojanovic Alexander Sasha
• Savolainen Douglas C
• Kreider Mark

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06F16/2219

  Large Object storage; Management thereof · CPC title

• G06F16/9038

  Presentation of query results · CPC title

• G06F16/211

  Schema design and management · CPC title