Data analysis system and method

Patent metadata

Field	Value
Publication number	US-10838970-B2
Application number	US-201816028886-A
Country	US
Kind code	B2
Filing date	Jul 6, 2018
Priority date	Oct 5, 2017
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

This disclosure relates to a system and method for data analysis. According to a first aspect, there is described a method, the method being performed using one or more processors, comprising: receiving one or more user inputs indicative of one or more relationships between data in a plurality of datasets; determining, based on the one or more user inputs, at least one object view for visualising the data in the plurality of datasets; generating, based on the one or more user inputs, metadata comprising: an object graph indicative of the one or more relationships between two or more of the plurality of datasets; and information identifying the at least one object view; and in response to a query relating to the plurality of datasets, using the metadata to determine how response data responding to the query should be provided.

First claim

What is claimed is:

1. A method, the method being performed using one or more processors, comprising:
accessing a plurality of datasets, wherein the plurality of datasets includes tables such that each table comprises rows representing objects and columns representing object properties;
receiving one or more user inputs indicative of one or more relationships between data in the plurality of datasets;
determining, based at least in part on the one or more user inputs, at least one object view for visualizing the data in the plurality of datasets;
determining a global property associated with a portion of data in at least two of the plurality of datasets, wherein the global property indicates one or more columns representing object properties that are associated with the portion of data;
generating, based at least in part on the one or more user inputs, metadata comprising:
an object graph indicative of the one or more relationships between two or more of the plurality of datasets;
the global property; and
information identifying the at least one object view; and

in response to a query relating to the plurality of datasets, using the metadata to determine how response data responding to the query should be provided, wherein the query is guided by the global property such that the query is applied to the portion of data associated with the global property.

2. The method of claim 1 , wherein the metadata further comprises a join between at least two datasets.

3. The method of claim 2 , wherein the global property is assigned to a portion of the data in at least two of the plurality of datasets.

4. The method of claim 3 , wherein determining how the response data should be provided comprises searching the plurality of datasets, wherein the search is guided by the global property.

5. The method of claim 4 , wherein the user input defines a global property that applies to the plurality of datasets.

6. The method of claim 5 , wherein the global property is determined using a method comprising:
receiving one or more statistics relating to the dataset;
comparing the one or more statistics to one or more threshold criteria; and
in response to one or more of the threshold criteria being met, providing a suggested global property to a user associated with the user input.

7. The method of claim 6 , wherein the user input comprises confirming the suggested global property provided to the user.

8. The method of claim 7 , wherein the one or more thresholds comprises at least one of: a proportion of datasets comprising an object type or property; a uniqueness score; cardinalities of object types or properties; or a title key.

9. The method of claim 8 , wherein the method further comprises inferring one or more joins between datasets based on the global property.

10. The method of claim 9 , wherein the global property is defined by one or more data formats or one or more data types.

11. The method of claim 10 , further comprising synchronizing the metadata with a metadata layer stored on a server.

12. The method of claim 11 , wherein the at least one object view comprises an object prominence.

13. A system comprising:
memory configured to store instructions; and
one or more processors in communication with the memory, the one or more processors configured to execute the instructions to cause the system to:
access a plurality of datasets, wherein the plurality of datasets includes tables such that each table comprises rows representing objects and columns representing object properties;
receive, via a user interface, one or more user inputs indicative of one or more relationships between data in the plurality of datasets;
determine, based at least in part on the one or more user inputs, at least one object view for visualizing the data in the plurality of datasets;
determining a global property associated with a portion of data in at least two of the plurality of datasets, wherein the global property indicates one or more columns representing object properties that are associated with the portion of data;
generate, based at least in part on the one or more user inputs, metadata comprising:
an object graph indicative of the one or more relationships between two or more of the plurality of datasets;
the global property; and
information identifying the at least one object view; and

in response to a query relating to the plurality of datasets, use the metadata to determine how response data responding to the query should be provided wherein the query is guided by the global property such that the query is applied to the portion of data associated with the global property.

14. The system of claim 13 , wherein the metadata further comprises a join between at least two datasets.

15. The system of claim 14 , wherein the global property is assigned to a portion of the data in at least two of the plurality of datasets.

16. The system of claim 15 , wherein determining how the response data should be provided comprises searching the plurality of datasets, wherein the search is guided by the global property.

17. The system of claim 16 , wherein the user input defines a global property that applies to the plurality of datasets.

18. The system of claim 17 , wherein the one or more processors are configured to execute the instructions to further cause the system to determine the global property by at least:
receiving one or more statistics relating to the dataset;
comparing the one or more statistics to one or more threshold criteria; and
in response to one or more of the threshold criteria being met, providing a suggested global property to a user associated with the user input.

19. The system of claim 18 , wherein the user input comprises confirming the suggested global property provided to the user.

20. A non-transitory computer readable medium having computer readable code stored thereon, the computer readable code, when executed by at least one processor, causing performance of the steps of:
accessing a plurality of datasets, wherein the plurality of datasets includes tables such that each table comprises rows representing objects and columns representing object properties;
receiving, via a user interface, one or more user inputs indicative of one or more relationships between data in the plurality of datasets;
determining, based at least in part on the one or more user inputs, at least one object view for visualizing the data in the plurality of datasets;
determining a global property associated with a portion of data in at least two of the plurality of datasets, wherein the global property indicates one or more columns representing object properties that are associated with the portion of data;
generating, based at least in part on the one or more user inputs, metadata comprising:
an object graph indicative of the one or more relationships between two or more of the plurality of datasets;
the global property; and
information identifying the at least one object view; and

in response to a query relating to the plurality of datasets, using the metadata to determine how response data responding to the query should be provided, wherein the query is guided by the global property such that the query is applied to the portion of data associated with the global property.

Assignees

• Palantir Technologies Inc

Inventors

• Scolnick Myles
• Grossman Jack
• Inoue Jim

Classifications

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/367

  Ontology · CPC title

• G06F16/248

  Presentation of query results · CPC title

• G06F16/2456

  Join operations · CPC title

• G06F16/2462 Primary

  Approximate or statistical queries · CPC title